Well I did that (and built pythonnet from source) and everything seems to go fine until I try to import pytest in my C# code -> PythonEngine.ImportModule("pytest"); which results in:
#19 4.234 Unhandled exception. Python.Runtime.PythonException: ImportError : /usr/lib/python3.8/lib-dynload/_contextvars.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyContextVar_Type
#19 4.260 [' File "/usr/local/lib/python3.8/dist-packages/pytest/init.py", line 42, in <module>\n from _pytest.python_api import approx\n', ' File "/usr/local/lib/python3.8/dist-packages/_pytest/python_api.py", line 6, in <module>\n from decimal import Decimal\n', ' File "/usr/lib/python3.8/decimal.py", line 8, in <module>\n from _pydecimal import *\n', ' File "/usr/lib/python3.8/_pydecimal.py", line 440, in <module>\n import contextvars\n', ' File "/usr/lib/python3.8/contextvars.py", line 1, in <module>\n from _contextvars import Context, ContextVar, Token, copy_context\n'] at Python.Runtime.PythonException.ThrowIfIsNull(IntPtr ob) in /pythonnet/src/runtime/pythonexception.cs:line 261
#19 4.260 at Python.Runtime.PythonEngine.ImportModule(String name) in /pythonnet/src/runtime/pythonengine.cs:line 493
#19 4.260 at PythonNetIntegration.Program.Main(String[] args) in /source/PythonXunitTests.cs:line 119
This is my dockerfile. Apologies in advance for spaghetti code and lots of bodge this is something like attempt #53:
FROM mcr.microsoft.com/dotnet/sdk:5.0-focal-amd64 AS build
ENV DEBIAN_FRONTEND noninteractive
RUN : \
&& apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
python3 python3-pip python3-dev python3-venv
RUN python3 --version
RUN git clone https://github.com/pythonnet/pythonnet
WORKDIR /pythonnet
RUN python3 setup.py build_dotnet --inplace
ADD deps /deps
RUN python3 -m pip install -r /pythonnet/requirements.txt
RUN which pytest
WORKDIR /source
COPY ./PyTestSelenium/ /pythonCode/
COPY *.csproj .
RUN dotnet restore
COPY . .
RUN dotnet run
I would like to learn more about the internals of pythonnet, especially how you guyes do the calls from Python to .NET; Initially, I thought you are probably using the "Mono Embedding" (https://www.mono-project.com/docs/advanced/embedding/) via ctypes and creating a nice wrapper around it.
But I wasnt able to find much in the pytonnet repo - maybe someone could give me a hint?
The background why I'm asking: it is reported that with .NET 6, Mono will be discontinued - so, will .NET 5 and .NET 6 be a major change for pythonnet?
Thanks a lot in advance,
Florian
if (!PythonEngine.IsInitialized)
PythonEngine.Initialize();
Py.Import("keras"); // throws hereSystem.IO.FileNotFoundException: {"Could not load file or assembly 'org, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.":"org, Culture=neutral, PublicKeyToken=null"}
can someone tell me how to use python.net in my c# project.
http://pythonnet.github.io/ That could help.
import torch
import torch.nn as nn
import numpy as np
import torch.optim as optim
from matplotlib import pyplot as plt
input_size = 1
batch_size = 1
hidden_size = 16
num_layers = 1
output_size = 1
class Net(nn.Module):
def __init__(self):
super().__init__()
self.rnn = nn.RNN(
input_size=input_size, # feature_len = 1
hidden_size=hidden_size, # 隐藏记忆单元个数hidden_len = 16
num_layers=num_layers, # 网络层数 = 1
batch_first=True # 在传入数据时,按照[batch,seq_len,feature_len]的格式
)
for p in self.rnn.parameters(): # 对RNN层的参数做初始化
nn.init.normal_(p, mean=0.0, std=0.001)
self.linear = nn.Linear(hidden_size, output_size) # 输出层
def forward(self, x, hidden_prev):
"""
x：一次性输入所有样本所有时刻的值(batch,seq_len,feature_len)
hidden_prev：第一个时刻空间上所有层的记忆单元(batch, num_layer, hidden_len)
输出out(batch,seq_len,hidden_len) 和 hidden_prev(batch,num_layer,hidden_len)
"""
out, hidden_prev = self.rnn(x, hidden_prev)
# 因为要把输出传给线性层处理，这里将batch和seq_len维度打平
# 再把batch=1添加到最前面的维度（为了和y做MSE）
# [batch=1,seq_len,hidden_len]->[seq_len,hidden_len]
out = out.view(-1, hidden_size)
#[seq_len,hidden_len]->[seq_len,output_size=1]
out = self.linear(out)
# [seq_len,output_size=1]->[batch=1,seq_len,output_size=1]
out = out.unsqueeze(dim=0)
return out, hidden_prev训练过程
learning_rate = 0.01
model = Net()
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
hidden_prev = torch.zeros(batch_size, num_layers, hidden_size) # 初始化记忆单元h0[batch,num_layer,hidden_len]
num_time_steps = 50 # 区间内取多少样本点
for iter in range(6000):
start = np.random.randint(3, size=1)[0] # 在0~3之间随机取开始的时刻点
time_steps = np.linspace(start, start + 10, num_time_steps) # 在[start,start+10]区间均匀地取num_points个点
data = np.sin(time_steps)
data = data.reshape(num_time_step, 1) # [num_time_steps,] -> [num_points,1]
# 输入前49个点(seq_len=49)，即下标0~48 [batch, seq_len, feature_len]
x = torch.tensor(data[:-1]).float().view(1, num_time_steps - 1, 1)
# 预测后49个点，即下标1~49
y = torch.tensor(data[1:]).float().view(1, num_time_steps - 1, 1)
# 以上步骤生成(x,y)数据对
output, hidden_prev = model(x, hidden_prev) # 喂入模型得到输出
hidden_prev = hidden_prev.detach() # at
loss = criterion(output, y) # 计算MSE损失
model.zero_grad()
loss.backward()
optimizer.step()
if iter % 1000 == 0:
print("Iteration: {} loss {}".format(iter, loss.item()))测试过程
先用同样的方式生成一组数据x,y
start = np.random.randint(3, size=1)[0]
time_steps = np.linspace(start, start + 10, num_time_steps)
data = np.sin(time_steps)
data = data.reshape(num_time_steps, 1)
x = torch.tensor(data[:-1]).float().view(1, num_time_steps - 1, 1)
y = torch.tensor(data[1:]).float().view(1, num_time_steps - 1, 1)
predictions = []
input = x[:, 0, :] # 取seqlen里面第0号数据
input = input.view(1, 1, 1) # input：[1,1,1]
for in range(x.shape[1]): # 迭代seq_len次
pred, hidden_prev = model(input, hidden_prev)
input = pred # 预测出的(下一个点的)序列pred当成输入(或者直接写成input, hidden_prev = model(input, hidden_prev))
predictions.append(pred.detach().numpy().ravel()[0])x = x.data.numpy()
y = y.data.numpy()
plt.plot(time_steps[:-1], x.ravel())
plt.scatter(time_steps[:-1], x.ravel(), c='r') # x值
plt.scatter(time_steps[1:], y.ravel(), c='y') # y值
plt.scatter(time_steps[1:], predictions, c='b') # y的预测值
plt.show()
pythonnet for QuickOPC, but I see that currently Python 3.9 is not supported. Is there a plan for Python 3.9 support? And if so, when?Thanks for your help in advance.
Hi. After exploring IronPython, and being stuck with it + numpy, I explore Pythonnet alternative.
I try a simple example C# app embedding python file *.py execution for start, but I'm surprised there is no wrapper for file processing in pythonnet.
Is there a reason
PyRun_FileExFlags(FILE fp, const char filename, int start, PyObject globals, PyObject locals, int closeit, PyCompilerFlags *flags)
isn't available ?
I'm running PythonNet from a Blazor project, when I run the project using IIS, pythonnet works as expected. When running the Blazor app using Kestrel PythonNet no longer works. Any ideas why this could be?
You can check if the environment variable set in IIS and kestrel are different.
Well After testing and trying pythonnet a little bit, here I am :
///Kept all my empirical previous testing in comment
class Program
{
static string _scriptsPath = @"C:\.....\ConsoleApp_TestPythonNet\Scripts";
static string[] _pyExts = { ".py" };
static void Main(string[] args)
{
Console.WriteLine("In C#");
//string before = Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Process);
//Console.WriteLine("PATH (process): " + before);
PythonSetupEnv.AddModulePath(_scriptsPath);
PythonSetupEnv.UpdateEnvPaths();
//Console.WriteLine("------------------------------------");
//string after = Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Process);
//Console.WriteLine("PATH (process): " + after);
//Environment.SetEnvironmentVariable("PYTHONPATH", _scriptsPath, EnvironmentVariableTarget.Process);
//var bf = before.Split(Path.PathSeparator);
//var af = after.Split(Path.PathSeparator);
//string[] res;
//if (bf.Length > af.Length)
// res = bf.Except(af).ToArray();
//else
// res = af.Except(bf).ToArray();
//Console.WriteLine("------------------------------------");
//foreach (var p in res)
// Console.WriteLine(" - " + p);
//Console.ReadKey();
using (Py.GIL())
{
//Console.WriteLine("PYTHONPATH (process): " + Environment.GetEnvironmentVariable("PYTHONPATH", EnvironmentVariableTarget.Process));
//PythonSetupEnv.AddModulePath(_scriptsPath);
//PythonSetupEnv.UpdateEnvPaths();
//PythonEngine.RunSimpleString("import sys;print(sys.path)");
//Console.ReadKey();
//Console.WriteLine("PYTHONPATH (process): " + Environment.GetEnvironmentVariable("PYTHONPATH", EnvironmentVariableTarget.Process));
//PythonEngine.RunSimpleString("import sys;print(sys.path)");
//Console.ReadKey();
//PythonEngine.Exec(FileUtilities.GetStringCodeFromFile(_filepath, _pyExts));
PythonEngine.Initialize();
using (PyScope scope = Py.CreateScope())
{
string code = FileUtilities.GetStringCodeFromFile(_scriptsPath+@"\poc.py", _pyExts);
var scriptCompiled = PythonEngine.Compile(code);
scope.Execute(scriptCompiled);
//dynamic pocPy = Py.Import("poc");
//dynamic pocPy = scope.Import("poc");
//var pocPy = PythonEngine.ImportModule("poc");
//scope.Execute(pocPy);
dynamic func = scope.Get("PrintInput");
dynamic res = func();
dynamic inputText = scope.Get("InputText");
Console.WriteLine("Inside C# code : "+inputText.ToString());
}
}
Console.ReadKey();
}
}
}class FileUtilities
{
public static string GetStringCodeFromFile(string filepath, string[] ext)
{
/// if we have an existing file with a correct extension
if (!string.IsNullOrEmpty(filepath) && File.Exists(filepath) && ext != null && ext.Length > 0 && ext.Contains(Path.GetExtension(filepath)))
return File.ReadAllText(filepath);
return null;
}
}public class PythonSetupEnv
{
private static Dictionary<string, List<string>> _pythonEnvPaths;
private static string _pythonPath;
public static Dictionary<string, List<string>> PythonEnvPaths
{
get
{
if (_pythonEnvPaths == null || _pythonEnvPaths.Count < 3)
_pythonEnvPaths = InitEnvPathsDict(_pythonPath);
return _pythonEnvPaths;
}
set => _pythonEnvPaths = value;
}
public static string PythonPath { get => _pythonPath; set => _pythonPath = value; }
private static string KeysToString()
{
string result = "";
foreach (var key in PythonEnvPaths.Keys)
result += (key + " ");
return result;
}
private static List<string> ParseEnvPaths(string rawPaths, char separator)
{
if (rawPaths != null)
return rawPaths.Split(new char[1] { separator }).ToList();
else
return new List<string>();
}
/// <summary>
/// Standard Environment Variable Initialisation, from a given Python directory path, or not
/// </summary>
/// <param name="path">The path to the pythyon directory</param>
/// <returns></returns>
private static Dictionary<string, List<string>> InitEnvPathsDict(string path = null)
{
var alreadyDefinedPATH = ParseEnvPaths(Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine), Path.PathSeparator);
alreadyDefinedPATH = alreadyDefinedPATH.Union(ParseEnvPaths(Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Process), Path.PathSeparator)).ToList();
alreadyDefinedPATH = alreadyDefinedPATH.Union(ParseEnvPaths(Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.User), Path.PathSeparator)).ToList();
var envDict = new Dictionary<string, List<string>>() {
{ "PATH", alreadyDefinedPATH },
{ "PYTHONHOME", ParseEnvPaths(Environment.GetEnvironmentVariable("PYTHONHOME", EnvironmentVariableTarget.Machine), Path.PathSeparator) },
{ "PYTHONPATH", ParseEnvPaths(Environment.GetEnvironmentVariable("PYTHONPATH", EnvironmentVariableTarget.Machine), Path.PathSeparator) }
};
if (path != null && Directory.Exists(path))
{
try
{
//PATH
AddEnvPath("PATH", path, envDict);
//PYTHONHOME
AddEnvPath("PYTHONHOME", path, envDict);
//PYTHONPATH
AddEnvPath("PYTHONPATH", Path.Combine(path, "Lib"), envDict);
AddEnvPath("PYTHONPATH", Path.Combine(path, "Lib", "site-packages"), envDict);
AddEnvPath("PYTHONPATH", Path.Combine(path, "DLLs"), envDict);
}
catch (Exception e) { Console.WriteLine(e.Message); }
}
return envDict;
}
/// <summary>
/// Add a custom module | script to look for
/// </summary>
/// <param name="path">The path to the script, library / directory</param>
public static void AddModulePath(string path)
{
try
{
AddEnvPath("PYTHONPATH", path);
}
catch(Exception e) { Console.WriteLine(e.Message); }
}
...... --><-- ......
public static void AddEnvPath(string envKey, string envPath, Dictionary<string, List<string>> dict = null)
{
if (dict == null)
dict = @PythonEnvPaths;
if (!Directory.Exists(envPath))
throw new DirectoryNotFoundException("Incorect path to this directory : " + envPath);
if (!dict.ContainsKey(envKey))
throw new KeyNotFoundException(envKey + " is not a correct environment key. Allowed ones are : " + KeysToString());
if (!dict[envKey].Contains(envPath)) dict[envKey].Add(envPath);
}
public static void UpdateEnvPaths()
{
string envPath = null;
foreach (var env in PythonEnvPaths)
{
//foreach (var path in env.Value.FindAll(x => Directory.Exists(x)))
foreach (var path in env.Value)
{
envPath = envPath + Path.PathSeparator.ToString() + path;
}
//trim first path separator
if (!string.IsNullOrEmpty(envPath))
envPath = envPath.Substring(1);
Environment.SetEnvironmentVariable(env.Key, envPath, EnvironmentVariableTarget.Process);
envPath = null;
}
}
}#poc.py
print("Inside Python poc.py")
InputText = "Python Text"
def PrintInput():
print("inside poc.py PrintInput function : " + InputText)
#PrintInput()So As you can see I tried
- to loop up my script using PYTHONPATH environment variable setup before (that worked)
- executing that script using
dynamic pocPy = Py.Import("poc") - calling a python func from C# usign
pocPy.PrintInput()(that worked)
But now when I want to "play" with my python variables / function from C# usign dynamic pocPy = Py.Import("poc") doesn't help me.
After looking on this I found some code example leading me to my current state :
PythonEngine.Initialize();
using (PyScope scope = Py.CreateScope())
{
string code = FileUtilities.GetStringCodeFromFile(_scriptsPath+@"\poc.py", _pyExts);
var scriptCompiled = PythonEngine.Compile(code);
scope.Execute(scriptCompiled);
dynamic func = scope.Get("PrintInput");
dynamic res = func();
dynamic inputText = scope.Get("InputText");
Console.WriteLine("Inside C# code : "+inputText.ToString());
}Doesn't it be possible to play with "scope" variable like I did finally, but using
Py.Import() (or something similar involving scope I suppose) ?