I'm still not entirely sure what's going on with the ClassDescriptorHelper and DebugHelper classes but assuming that you want to automatically add a logger to every class, I'd look at annotations/decorators.

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> def add_log(cls):
...     setattr(cls, 'dbg', logging.getLogger(cls.__name__))
...     return cls
... 
>>> @add_log
... class LongClassName:
...     def test_info(self):
...         self.dbg.info("helloWorld")
>>> x = LongClassName()
>>> x.test_info()
INFO:LongClassName:helloWorld
>>>

If you try to access an attribute of an instance (such as self.dbg), it will first look at its own instance variables, if it doesn't find one it'll look at the class's variable. If it doesn't find it there either it'll keep looking up the inheritance tree (slightly different between old and new style classes).

Alright, actually my question is rather stupid. After doing:

sc.addFile("/[AbsolutePathTo]/Test.py")

I still have to import the Test.py file like I would import a regular python file with:

import Test

then I can call

Test.func()

and it works. I thought that the "import Test" is not necessary since I add the file to the spark context, but apparently that does not have the same effect. Thanks mark91 for pointing me into the right direction.

UPDATE 28.10.2017:

as asked in the comments, here more details on the app.py

from pyspark import SparkContext
from pyspark.conf import SparkConf

conf = SparkConf()
conf.setMaster("local[4]")
conf.setAppName("Spark Stream")
sc = SparkContext(conf=conf)
sc.addFile("Test.py")

import Test

Test.func()

The shipped junitxml plugin does not have hooks to add such data you can print it to stdout though, since that gets added to the junitxml data.

So as long as you print out logs you will at least be able to know the data.

The simplest would probably be to increase the scope of your fixtures, the granularity is a bit limited (class, module, package and session) but if it works....

A related alternative is to have your function fixture depend on a larger "cache" fixture and inspect the request, that way it can check the properties of the requester and either create a new value or get one from the cache.

Finally, the "big hammer" is to dynamically generate fixture result by plugging into pytest_fixture_setup or pytest_generate_tests, something like that, and then do whatever you want on the fly.

Short Answer

It's boilerplate code that protects users from accidentally invoking the script when they didn't intend to. Here are some common problems when the guard is omitted from a script:

• If you import the guardless script in another script (e.g. import my_script_without_a_name_eq_main_guard), then the latter script will trigger the former to run at import time and using the second script's command line arguments. This is almost always a mistake.

• If you have a custom class in the guardless script and save it to a pickle file, then unpickling it in another script will trigger an import of the guardless script, with the same problems outlined in the previous bullet.

Long Answer

To better understand why and how this matters, we need to take a step back to understand how Python initializes scripts and how this interacts with its module import mechanism.

Whenever the Python interpreter reads a source file, it does two things:

• it sets a few special variables like __name__, and then

• it executes all of the code found in the file.

Let's see how this works and how it relates to your question about the __name__ checks we always see in Python scripts.

Code Sample

Let's use a slightly different code sample to explore how imports and scripts work. Suppose the following is in a file called foo.py.

# Suppose this is foo.py.

print("before import")
import math

print("before function_a")
def function_a():
    print("Function A")

print("before function_b")
def function_b():
    print("Function B {}".format(math.sqrt(100)))

print("before __name__ guard")
if __name__ == '__main__':
    function_a()
    function_b()
print("after __name__ guard")

Special Variables

When the Python interpreter reads a source file, it first defines a few special variables. In this case, we care about the __name__ variable.

When Your Module Is the Main Program

If you are running your module (the source file) as the main program, e.g.

python foo.py

the interpreter will assign the hard-coded string "__main__" to the __name__ variable, i.e.

# It's as if the interpreter inserts this at the top
# of your module when run as the main program.
__name__ = "__main__" 

When Your Module Is Imported By Another

On the other hand, suppose some other module is the main program and it imports your module. This means there's a statement like this in the main program, or in some other module the main program imports:

# Suppose this is in some other main program.
import foo

The interpreter will search for your foo.py file (along with searching for a few other variants), and prior to executing that module, it will assign the name "foo" from the import statement to the __name__ variable, i.e.

# It's as if the interpreter inserts this at the top
# of your module when it's imported from another module.
__name__ = "foo"

Executing the Module's Code

After the special variables are set up, the interpreter executes all the code in the module, one statement at a time. You may want to open another window on the side with the code sample so you can follow along with this explanation.

Always

1. It prints the string "before import" (without quotes).

2. It loads the math module and assigns it to a variable called math. This is equivalent to replacing import math with the following (note that __import__ is a low-level function in Python that takes a string and triggers the actual import):

# Find and load a module given its string name, "math",
# then assign it to a local variable called math.
math = __import__("math")

3. It prints the string "before function_a".

4. It executes the def block, creating a function object, then assigning that function object to a variable called function_a.

5. It prints the string "before function_b".

6. It executes the second def block, creating another function object, then assigning it to a variable called function_b.

7. It prints the string "before __name__ guard".

Only When Your Module Is the Main Program

8. If your module is the main program, then it will see that __name__ was indeed set to "__main__" and it calls the two functions, printing the strings "Function A" and "Function B 10.0".

Only When Your Module Is Imported by Another

8. (instead) If your module is not the main program but was imported by another one, then __name__ will be "foo", not "__main__", and it'll skip the body of the if statement.

Always

9. It will print the string "after __name__ guard" in both situations.

Summary

In summary, here's what'd be printed in the two cases:

# What gets printed if foo is the main program
before import
before function_a
before function_b
before __name__ guard
Function A
Function B 10.0
after __name__ guard

# What gets printed if foo is imported as a regular module
before import
before function_a
before function_b
before __name__ guard
after __name__ guard

Why Does It Work This Way?

You might naturally wonder why anybody would want this. Well, sometimes you want to write a .py file that can be both used by other programs and/or modules as a module, and can also be run as the main program itself. Examples:

• Your module is a library, but you want to have a script mode where it runs some unit tests or a demo.

• Your module is only used as a main program, but it has some unit tests, and the testing framework works by importing .py files like your script and running special test functions. You don't want it to try running the script just because it's importing the module.

• Your module is mostly used as a main program, but it also provides a programmer-friendly API for advanced users.

Beyond those examples, it's elegant that running a script in Python is just setting up a few magic variables and importing the script. "Running" the script is a side effect of importing the script's module.

Food for Thought

• Question: Can I have multiple __name__ checking blocks? Answer: it's strange to do so, but the language won't stop you.

• Suppose the following is in foo2.py. What happens if you say python foo2.py on the command-line? Why?

# Suppose this is foo2.py.
import os, sys; sys.path.insert(0, os.path.dirname(__file__)) # needed for some interpreters

def function_a():
    print("a1")
    from foo2 import function_b
    print("a2")
    function_b()
    print("a3")

def function_b():
    print("b")

print("t1")
if __name__ == "__main__":
    print("m1")
    function_a()
    print("m2")
print("t2")
      

• Now, figure out what will happen if you remove the __name__ check in foo3.py:

# Suppose this is foo3.py.
import os, sys; sys.path.insert(0, os.path.dirname(__file__)) # needed for some interpreters

def function_a():
    print("a1")
    from foo3 import function_b
    print("a2")
    function_b()
    print("a3")

def function_b():
    print("b")

print("t1")
print("m1")
function_a()
print("m2")
print("t2")

• What will this do when used as a script? When imported as a module?

# Suppose this is in foo4.py
__name__ = "__main__"

def bar():
    print("bar")
    
print("before __name__ guard")
if __name__ == "__main__":
    bar()
print("after __name__ guard")

Tired of sys.path hacks?

There are plenty of sys.path.append -hacks available, but I found an alternative way of solving the problem in hand.

Summary

• Wrap the code into one folder (e.g. packaged_stuff)
• Create setup.py script where you use setuptools.setup(). (see minimal setup.py below)
• Pip install the package in editable state with pip install -e <myproject_folder>
• Import using from packaged_stuff.modulename import function_name

---

Setup

The starting point is the file structure you have provided, wrapped in a folder called myproject.

.
└── myproject
    ├── api
    │   ├── api_key.py
    │   ├── api.py
    │   └── __init__.py
    ├── examples
    │   ├── example_one.py
    │   ├── example_two.py
    │   └── __init__.py
    ├── LICENCE.md
    ├── README.md
    └── tests
        ├── __init__.py
        └── test_one.py

I will call the . the root folder, and in my example case it is located at C:\tmp\test_imports\.

api.py

As a test case, let's use the following ./api/api.py

def function_from_api():
    return 'I am the return value from api.api!'

test_one.py

from api.api import function_from_api

def test_function():
    print(function_from_api())

if __name__ == '__main__':
    test_function()

Try to run test_one:

PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
  File ".\myproject\tests\test_one.py", line 1, in <module>
    from api.api import function_from_api
ModuleNotFoundError: No module named 'api'

Also trying relative imports wont work:

Using from ..api.api import function_from_api would result into

PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
  File ".\tests\test_one.py", line 1, in <module>
    from ..api.api import function_from_api
ValueError: attempted relative import beyond top-level package

---

Steps

1. Make a setup.py file to the root level directory

---

The contents for the setup.py would be*

from setuptools import setup, find_packages

setup(name='myproject', version='1.0', packages=find_packages())

2. Use a virtual environment

---

If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder)

• Create virtual env
  • python -m venv venv
• Activate virtual env
  • source ./venv/bin/activate (Linux, macOS) or ./venv/Scripts/activate (Win)

To learn more about this, just Google out "python virtual env tutorial" or similar. You probably never need any other commands than creating, activating and deactivating.

Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis

PS C:\tmp\test_imports> python -m venv venv
PS C:\tmp\test_imports> .\venv\Scripts\activate
(venv) PS C:\tmp\test_imports>

and your folder tree should look like this**

.
├── myproject
│   ├── api
│   │   ├── api_key.py
│   │   ├── api.py
│   │   └── __init__.py
│   ├── examples
│   │   ├── example_one.py
│   │   ├── example_two.py
│   │   └── __init__.py
│   ├── LICENCE.md
│   ├── README.md
│   └── tests
│       ├── __init__.py
│       └── test_one.py
├── setup.py
└── venv
    ├── Include
    ├── Lib
    ├── pyvenv.cfg
    └── Scripts [87 entries exceeds filelimit, not opening dir]

3. pip install your project in editable state

---

Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package.

In the root directory, run

pip install -e . (note the dot, it stands for "current directory")

You can also see that it is installed by using pip freeze

(venv) PS C:\tmp\test_imports> pip install -e .
Obtaining file:///C:/tmp/test_imports
Installing collected packages: myproject
  Running setup.py develop for myproject
Successfully installed myproject
(venv) PS C:\tmp\test_imports> pip freeze
myproject==1.0

4. Add myproject. into your imports

---

Note that you will have to add myproject. only into imports that would not work otherwise. Imports that worked without the setup.py & pip install will work still work fine. See an example below.

---

Test the solution

Now, let's test the solution using api.py defined above, and test_one.py defined below.

test_one.py

from myproject.api.api import function_from_api

def test_function():
    print(function_from_api())

if __name__ == '__main__':
    test_function()

running the test

(venv) PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
I am the return value from api.api!

---

* See the setuptools docs for more verbose setup.py examples.

** In reality, you could put your virtual environment anywhere on your hard disk.

Check out my project johnnydep!

Installation:

pip install johnnydep

Usage example:

$ johnnydep requests
name                       summary
-------------------------  ----------------------------------------------------------------------
requests                   Python HTTP for Humans.
├── certifi>=2017.4.17     Python package for providing Mozilla's CA Bundle.
├── chardet<3.1.0,>=3.0.2  Universal encoding detector for Python 2 and 3
├── idna<2.7,>=2.5         Internationalized Domain Names in Applications (IDNA)
└── urllib3<1.23,>=1.21.1  HTTP library with thread-safe connection pooling, file post, and more.

A more complex tree:

$ johnnydep ipython 
name                              summary
--------------------------------  -----------------------------------------------------------------------------
ipython                           IPython: Productive Interactive Computing
├── appnope                       Disable App Nap on OS X 10.9
├── decorator                     Better living through Python with decorators
├── jedi>=0.10                    An autocompletion tool for Python that can be used for text editors.
│   └── parso==0.1.1              A Python Parser
├── pexpect                       Pexpect allows easy control of interactive console applications.
│   └── ptyprocess>=0.5           Run a subprocess in a pseudo terminal
├── pickleshare                   Tiny 'shelve'-like database with concurrency support
├── prompt-toolkit<2.0.0,>=1.0.4  Library for building powerful interactive command lines in Python
│   ├── six>=1.9.0                Python 2 and 3 compatibility utilities
│   └── wcwidth                   Measures number of Terminal column cells of wide-character codes
├── pygments                      Pygments is a syntax highlighting package written in Python.
├── setuptools>=18.5              Easily download, build, install, upgrade, and uninstall Python packages
├── simplegeneric>0.8             Simple generic functions (similar to Python's own len(), pickle.dump(), etc.)
└── traitlets>=4.2                Traitlets Python config system
    ├── decorator                 Better living through Python with decorators
    ├── ipython-genutils          Vestigial utilities from IPython
    └── six                       Python 2 and 3 compatibility utilities

I've never quite seen anything like this before and we've used network-X with compiled fortran wrapped in F2py, running under MPI many many times.

I suggest that you remove and re-install your network-x package.

Which python are you using and what os are you running on? We've had very good luck running the anaconda python installation. You have to be a bit careful when installing petsc though. Building from source and running the PETSc tests is the safest way.

Peewee models implement a get_id() method, which is used by the pk_expr() method. I am guessing that is the source of the issue.

Fixed in master: https://github.com/coleifer/peewee/issues/435