Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

embedding docs - expose all available API #410

Open
denfromufa opened this issue Feb 27, 2017 · 11 comments
Open

embedding docs - expose all available API #410

denfromufa opened this issue Feb 27, 2017 · 11 comments

Comments

@denfromufa
Copy link
Contributor

@denfromufa denfromufa commented Feb 27, 2017

cc @yagweb @vmuriart

Initialize
Py.GIL and Threads
RunSimpleString, Exec, Eval
Py.Import
Dynamic vs static declaration
Py.kw
Setting paths
Type conversions
Shutdown

@vmuriart
Copy link
Contributor

@vmuriart vmuriart commented Feb 27, 2017

Here's a list of whats currently publicly exposed but with no xml docs.

Python.Runtime.PythonDerivedType.InvokeMethodVoid(Python.Runtime.IPythonDerivedType, string, string, object[])
Python.Runtime.PythonDerivedType.InvokeGetProperty<T>(Python.Runtime.IPythonDerivedType, string)
Python.Runtime.PythonDerivedType.InvokeSetProperty<T>(Python.Runtime.IPythonDerivedType, string, T)
Python.Runtime.PythonDerivedType.InvokeCtor(Python.Runtime.IPythonDerivedType, string, object[])
Python.Runtime.PythonDerivedType.Finalize(Python.Runtime.IPythonDerivedType)
Python.Runtime.ConverterExtension
Python.Runtime.ConverterExtension.ToPython(this object)
Python.Runtime.Dispatcher
Python.Runtime.Dispatcher.target
Python.Runtime.Dispatcher.dtype
Python.Runtime.Dispatcher.Dispatcher(System.IntPtr, System.Type)
Python.Runtime.Dispatcher.~Dispatcher()
Python.Runtime.Dispatcher.Dispatch(System.Collections.ArrayList)
Python.Runtime.Dispatcher.TrueDispatch(System.Collections.ArrayList)
Python.Runtime.ConversionException
Python.Runtime.ConversionException.ConversionException()
Python.Runtime.ConversionException.ConversionException(string)
Python.Runtime.Exceptions.warn(string, System.IntPtr)
Python.Runtime.Exceptions.deprecation(string, int)
Python.Runtime.Exceptions.deprecation(string)
Python.Runtime.Exceptions.BaseException
Python.Runtime.Exceptions.Exception
Python.Runtime.Exceptions.StopIteration
Python.Runtime.Exceptions.GeneratorExit
Python.Runtime.Exceptions.StandardError
Python.Runtime.Exceptions.ArithmeticError
Python.Runtime.Exceptions.LookupError
Python.Runtime.Exceptions.AssertionError
Python.Runtime.Exceptions.AttributeError
Python.Runtime.Exceptions.EOFError
Python.Runtime.Exceptions.FloatingPointError
Python.Runtime.Exceptions.EnvironmentError
Python.Runtime.Exceptions.IOError
Python.Runtime.Exceptions.OSError
Python.Runtime.Exceptions.ImportError
Python.Runtime.Exceptions.IndexError
Python.Runtime.Exceptions.KeyError
Python.Runtime.Exceptions.KeyboardInterrupt
Python.Runtime.Exceptions.MemoryError
Python.Runtime.Exceptions.NameError
Python.Runtime.Exceptions.OverflowError
Python.Runtime.Exceptions.RuntimeError
Python.Runtime.Exceptions.NotImplementedError
Python.Runtime.Exceptions.SyntaxError
Python.Runtime.Exceptions.IndentationError
Python.Runtime.Exceptions.TabError
Python.Runtime.Exceptions.ReferenceError
Python.Runtime.Exceptions.SystemError
Python.Runtime.Exceptions.SystemExit
Python.Runtime.Exceptions.TypeError
Python.Runtime.Exceptions.UnboundLocalError
Python.Runtime.Exceptions.UnicodeError
Python.Runtime.Exceptions.UnicodeEncodeError
Python.Runtime.Exceptions.UnicodeDecodeError
Python.Runtime.Exceptions.UnicodeTranslateError
Python.Runtime.Exceptions.ValueError
Python.Runtime.Exceptions.ZeroDivisionError
Python.Runtime.Exceptions.Warning
Python.Runtime.Exceptions.UserWarning
Python.Runtime.Exceptions.DeprecationWarning
Python.Runtime.Exceptions.PendingDeprecationWarning
Python.Runtime.Exceptions.SyntaxWarning
Python.Runtime.Exceptions.RuntimeWarning
Python.Runtime.Exceptions.FutureWarning
Python.Runtime.Exceptions.ImportWarning
Python.Runtime.Exceptions.UnicodeWarning
Python.Runtime.DocStringAttribute.DocStringAttribute(string)
Python.Runtime.DocStringAttribute.DocString
Python.Runtime.INativeCall.Void_Call_0(System.IntPtr)
Python.Runtime.INativeCall.Void_Call_1(System.IntPtr, System.IntPtr)
Python.Runtime.INativeCall.Int_Call_3(System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr)
Python.Runtime.INativeCall.Call_3(System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr)
Python.Runtime.PyAnsiString
Python.Runtime.PyIter.Dispose(bool)
Python.Runtime.PyIter.MoveNext()
Python.Runtime.PyIter.Reset()
Python.Runtime.PyIter.Current
Python.Runtime.PyNumber.PyNumber(System.IntPtr)
Python.Runtime.PyNumber.PyNumber()
Python.Runtime.PyObject.obj
Python.Runtime.PyObject.PyObject()
Python.Runtime.PyObject.~PyObject()
Python.Runtime.PyObject.Dispose()
Python.Runtime.PyObject.Refcount
Python.Runtime.PyObject.TryGetMember(System.Dynamic.GetMemberBinder, out object)
Python.Runtime.PyObject.TrySetMember(System.Dynamic.SetMemberBinder, object)
Python.Runtime.PyObject.TryInvokeMember(System.Dynamic.InvokeMemberBinder, object[], out object)
Python.Runtime.PyObject.TryInvoke(System.Dynamic.InvokeBinder, object[], out object)
Python.Runtime.PyObject.TryConvert(System.Dynamic.ConvertBinder, out object)
Python.Runtime.PyObject.TryBinaryOperation(System.Dynamic.BinaryOperationBinder, object, out object)
Python.Runtime.PyObject.TryUnaryOperation(System.Dynamic.UnaryOperationBinder, out object)
Python.Runtime.PySequence.PySequence(System.IntPtr)
Python.Runtime.PySequence.PySequence()
Python.Runtime.PythonEngine.PythonEngine()
Python.Runtime.PythonEngine.PythonEngine(params string[])
Python.Runtime.PythonEngine.PythonEngine(System.Collections.Generic.IEnumerable<string>)
Python.Runtime.PythonEngine.Dispose()
Python.Runtime.PythonEngine.IsInitialized
Python.Runtime.PythonEngine.ProgramName
Python.Runtime.PythonEngine.PythonHome
Python.Runtime.PythonEngine.PythonPath
Python.Runtime.PythonEngine.Version
Python.Runtime.PythonEngine.BuildInfo
Python.Runtime.PythonEngine.Platform
Python.Runtime.PythonEngine.Copyright
Python.Runtime.PythonEngine.RunSimpleString(string)
Python.Runtime.PythonEngine.Initialize()
Python.Runtime.PythonEngine.Initialize(bool)
Python.Runtime.RunFlagType
Python.Runtime.RunFlagType.Single
Python.Runtime.RunFlagType.File
Python.Runtime.RunFlagType.Eval
Python.Runtime.Py
Python.Runtime.Py.GIL()
Python.Runtime.Py.GILState
Python.Runtime.Py.GILState.Dispose()
Python.Runtime.Py.GILState.~GILState()
Python.Runtime.Py.KeywordArguments
Python.Runtime.Py.kw(params object[])
Python.Runtime.Py.Import(string)
Python.Runtime.Py.SetArgv()
Python.Runtime.Py.SetArgv(params string[])
Python.Runtime.Py.SetArgv(System.Collections.Generic.IEnumerable<string>)
Python.Runtime.PythonException.PythonException()
Python.Runtime.PythonException.~PythonException()
Python.Runtime.Runtime.UCS
Python.Runtime.Runtime.pyversion
Python.Runtime.Runtime.pyversionnumber
Python.Runtime.Runtime.dll
Python.Runtime.Runtime.Py_Main(int, string[])

@vmuriart
Copy link
Contributor

@vmuriart vmuriart commented Feb 27, 2017

The easier approach might to write a few more example based on the unittests and limit those examples to use only API that will remain public.

@yagweb
Copy link
Contributor

@yagweb yagweb commented Feb 28, 2017

@denfromufa I'm not going very far in pythonnet. currently, I have used these methods,

  1. PythonEngine.PythonPath: to embed CPython into .NET as a independ module
  2. Initialize
  3. Py.GIL method:
  4. PyScope class: I use it instead of RunSimpleString, Exec and Eval
  5. PythonEngine.AcquireLock and PythonEngine.ReleaseLock: used in C# callback funtion passed to CPython
  6. PyIter class: to analyse the OrderedDict object return from CPython
  7. dynamic type: used for visiting Python object, field and method.
  8. KeywordArguments class: but did not use Py.kw method

In the future, I will dig deep, such as using threading and asynchronous callback, and share my experience here.

Before writing the docs, I think we need to do some work on the embedding API design.

For the embedding API design, I hold the same view as @filmor,

All functions that operate on pointers directly should be internal. one of the reasons is security.

According to this principle, some functions and properties can be removed from the current API list, such as the Handle property of PyObject, the BeginAllowThreads method, the EndAllowThreads.

In order to remove these APIs related to the IntPtr, some new APIs need to be designed and provided. Use the PythonEngine.AcquireLock and PythonEngine.ReleaseLock as example, these two methods should be removed according to this principle. Their problems include, programe will crash if the ReleaseLock is called redundantly, and CPython will be blocked if forge to call the ReleaseLock method. The Py.GIL class can be extended to cover their usage, such as the PyScope.GIL class in the PyScope proposal.

Following are my other proposals for the embedding API design,

  1. Merge the PythonEngine class and Py class. PythonEngine called the methods in Py such as Throw, Py called the PythonEngine in PythonEngine, I don‘t know the principle of where to put a method, Py or PythonEngine.
  2. The KeywordArguments class can be moved out from Py class, the Py.kw method can be put into the KeywordArguments class as a static convert method, and other convert methods can also be putted into the KeywordArguments as overloading methods. For example, I used a convert method to transform C# Dictionary into KeywordArguments.
  3. It seems there is no way to flatten a list and pass it to a python function. A new TupleArguments class similar to KeywordArguments can do this.
  4. The GILState class can be moved out from Py class, and should be extended as shown above.
  5. A class wrapping threading may be needed. (explore later)

For the docs of embedding CPython, @vmuriart said the easier approach is writing a few more example based on the unittests, supplementary approach is providing a doc according to topics. Topics come from the list of issues users encontered. For example, following is what I have currently,

  1. how to embed CPython into .NET as a independ module. I developed a python module to extract and pack the moduled into a zip file.
  2. Pass .NET object as attribute of CPython object. For example,
    how to redirect sys.stdout and sys.stdin
  3. Call python function
    how to call a python function of signature "def func(arg1, *arg2, **kwargs)"
  4. Transform CPython object (custom and build-in) into .NET Object. For example,
    how to convert a CPython OrderedDict object into C# Dictionary
  5. Callback
    how to pass CPython object to the callback function defined in .NET (I encontered a bug here)
@denfromufa
Copy link
Contributor Author

@denfromufa denfromufa commented Mar 1, 2017

here is a library for embedding python with pythonnet and using numpy, matplotlib:

https://github.com/Lodomir/PythonInterface

@JimSEOW
Copy link

@JimSEOW JimSEOW commented Apr 5, 2017

@denfromufa
Can you take a look why in PythonInterface
the pyplot.cs

Line 96 and 97 fail with the latest Python.Runtime.dll (compile from git clone)

        public void plot(double[] x, double[] y)
        {
            Python.PushLocal(Local);
            Python.Set("x", x);  //line96
            Python.Set("y", y);  //line97
            Python.RunString(
                "fig = plt.figure()\n" +
                "ax = fig.add_subplot(111)\n" +
                "ax.plot(x, y)\n" +
                "del x\n" +
                "del y\n"
                );
            Local = Python.PopLocal();
            UpdatePlot();
        }

using example

                double[] x = { 1.0, 2.0, 3.0, 4.0 };
                double[] y = { 1.0, 2.0, 1.0, 2.0 };
                this.pyplot.plot(x, y);
        public void Set(string name, dynamic content)
        {
            using (Py.GIL())
            {
                System.Type ValueType = content.GetType();
                // Arrays are always converted to python lists
                if (ValueType.IsArray)
                {
                    if (!NumpyLoaded)
                    {
                        throw new PythonException("numpy must be loaded for array conversion.");
                    }

                    // BlockCopy possibly multidimensional array of arbitrary type to onedimensional byte array
                    System.Type ElementType = ValueType.GetElementType();
                    int nbytes = content.Length * Marshal.SizeOf(ElementType);
                    byte[] data = new byte[nbytes];
                    System.Buffer.BlockCopy(content, 0, data, 0, nbytes);

                    // <=== HERE IS THE EFFOR===> Create an python tuple with the dimensions of the input array 
                    PyObject[] lengths = new PyObject[content.Rank];
                    for (int i = 0; i < content.Rank; i++)
                        lengths[i] = new PyInt(content.GetLength(i));
                    PyTuple shape = new PyTuple(lengths);

                    // Create an empty numpy array in correct shape and datatype
                    dynamic dtype;
                    if (ElementType == typeof(int))
                        dtype = np.int32;
                    else if (ElementType == typeof(double))
                        dtype = np.float64;
                    else
                        throw new System.Exception("Datatype not supported!");
                    dynamic pydata = np.empty(shape, dtype);

                    // Copy the data to that array
                    System.IntPtr ptr = (System.IntPtr)PyInt.AsInt(pydata.__array_interface__["data"][0]).ToInt32();
                    Marshal.Copy(data, 0, ptr, nbytes);

                    // Push the variable to local dictionary
                    UpdateLocals(name, pydata);
                }
                else
                {
                    if (ValueType == typeof(int))
                        UpdateLocals(name, new PyInt(content));
                    else if (ValueType == typeof(string))
                        UpdateLocals(name, new PyString(content));
                    else if (ValueType == typeof(double))
                        UpdateLocals(name, new PyFloat(content));
                    else if (ValueType == typeof(bool))
                    {
                        UpdateLocals(name, new PyInt(System.Convert.ToInt32(content)));
                        RunStringTry(name + " = (" + name + " == 1)");
                    }
                    else
                        throw new PythonException("Datatype not supported!");
                }
            }
        }

previously, I thought was due to regression of Numpy. With the latest compile python.Runtime, the regression issue is gone, yet, there is a problem of converting double[] array to python.net

Appreciate

@JimSEOW
Copy link

@JimSEOW JimSEOW commented Apr 5, 2017

PyPlot for matplotlib
#442

@denfromufa
Copy link
Contributor Author

@denfromufa denfromufa commented Apr 5, 2017

@JimSEOW
Copy link

@JimSEOW JimSEOW commented Apr 5, 2017

@denfromufa sorry, I am not aware.

@yagweb
Copy link
Contributor

@yagweb yagweb commented Apr 6, 2017

@JimSEOW I tried the SimplePythonGraph project. I guess you ran the program in x64 mode. If so, the bug you encountered was caused by the limitation of method PyInt.AsInt().

A solution is,

replace all the expressions PyInt.AsInt(...).ToInt32() and PyInt.AsInt(...).ToInt64() with (...).AsManagedObject(typeof(long)). They appear in both the Get() and Set() methods of PythonConnection class.

P.S.
Since AsManagedObject method is widely used and not very convenient, I am trying to add a new Generic method As<T> to the PyObject class, after that, one can use
(...).As<long>()
instead of
(...).AsManagedObject(typeof(long))

The limitation of the PyInt.AsInt method is,

This method called the CPython API PyNumber_Int, which converts a python number to C int (maybe C long?). When the target platform of the program was x64, the address of a variable may out of the range of integer. PyNumber_Long is also not working here, because on windows the length of C long is also 4 bytes, no matter the target platform is x86 or x64.

@JimSEOW
Copy link

@JimSEOW JimSEOW commented Apr 9, 2017

 public void Set(string name, dynamic content)  //Only 1 modification
//System.IntPtr ptr = (System.IntPtr)PyInt.AsInt(pydata.__array_interface__["data"][0]).ToInt32();
   System.IntPtr ptr = (System.IntPtr)pydata.__array_interface__["data"][0].AsManagedObject(typeof(long));

public dynamic Get(string name_in)  //only 2 modifications
 //System.IntPtr ptr = (System.IntPtr)PyInt.AsInt(Local.GetItem("__" + name + "__ptr")).ToInt32();
System.IntPtr ptr = (System.IntPtr)Local.GetItem("__" + name + "__ptr").AsManagedObject(typeof(long)); 

//int nbytes = PyInt.AsInt(Local.GetItem("__" + name + "__nbytes")).ToInt32();
 long nbytes = (long)Local.GetItem("__" + name + "__nbytes").AsManagedObject(typeof(long)); 

I get error of {"Specified cast is not valid."}
public dynamic Get(string name_in)

RunString("__" + name + "__dtype = str(" + name + ".dtype)"); <=========

@yagweb Do you have a running SimplePythonGraph project to share?

Relevant issue

@JimSEOW
Copy link

@JimSEOW JimSEOW commented Apr 27, 2017

@yagweb will your scope class address the GET, SET issues above?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants