Python Multithread/process combination of Json and Python?

Question

So currently im trying to get a function right. Basically im trying to do a multiple (Concurrently) thread that takes information from a Json file into the program and then for every thread it should use each json object and then execute the code with those information.

What I have done so far - This code is only for Multiprocess which indeed works.:

#Read json File
with open('profileMulti.json', 'r', encoding='UTF-8') as json_data:
    profiles_string = json.load(json_data)

def get_individual_profiles(config):
    top_layer = config.get('Profiles')
    if top_layer:
        top_level_keys = ['profile_{}'.format(i) for i in range(len(top_layer))]
        print(top_level_keys)
        return [(key, top_layer.get(key)) for key in top_level_keys]
    return []

def stringify(key, next_layer):
    return [
        ' '.join(key.capitalize().split('_')),
        next_layer.get('Name'),
        next_layer.get('Last_Name'),
        next_layer.get('Email'),
        next_layer.get('Phone')
        #etc etc...
        ]

    config = profiles_string
profiles = get_individual_profiles(config)

pool = ThreadPool()

# Launch a process for each item
threads = [pool.apply_async(stringify, tuple(item)) for item in profiles]


# get() the results as each finishes
results = [res.get() for res in threads]
print('threaded results:')
for item in results:
    print(item)

Output:

threaded results:
['Profile 0', 'Thrill', 'Ofit', '[email protected]', '123 412 123']

['Profile 1', 'Hellow', 'World', '[email protected]', '543 412 312']

Which is great. But the issues is:

That whenever I want to etc use those info

EXEMPLE

def checkoutNames(NameUrl, nameID):
payload = {
        "shared": {
            "challenge": {
              "Name": item["Name"],
              "Last_Name": item["Last_Name"],
              "Email": item["Email"],
              "Phone": item["Phone"],

Is that it wont recognize those attributes but what I need to do is that I need to call them by item1.. item[2] etc from that order that is from def stringify(key, next_layer): which I don't know by now how to make it so I don't need to do it.

The other problem is that whenever we say ETC that I use item1 in the code. Then it will only use the last thread and skip the rest. So if I do

print(item[1])

Then the only output that will come is the last one which is Hello

So the problems I need to get solved:

1. To make each thread to execute concurrently and execute the code with those information

2. Make a fix so I don't need to use item1 but to use instead item['Name'].

So the question is, is this possible and what is the ideas about this?

EDIT - This is the code i'm currently having with only one Json profile and that is working fine with just one profile. This is WITHOUT multiprocessing

Json file

      {
    "Profiles": {
        "profile_0": {
            "Url": "Myownwebsite.se",
            "My-Note": "Helloworld",
            "Email": "[email protected]"
            "PersonNumber": "1234543",
            "postal_code": "54123",
            "given_name": "World",
            "Last_name": "Hellow",
            "street_address": "helloworld 123",
            "city": "Stockholm",
            "country": "Sweden",
            "phone": "123456789",
            "Color": "Red",
            "house_number": "123",
            "year": "2017"
        },
        "profile_1": {
            "Url": "Myasdwfaesite.se",
            "My-Note": "aasfase",
            "Email": "[email protected]"
            "PersonNumber": "5634543",
            "postal_code": "123445",
            "given_name": "Balling",
            "Last_name": "Calling",
            "street_address": "qwertr 123",
            "city": "London",
            "country": "UK",
            "phone": "65412331",
            "Color": "Blue",
            "house_number": "321",
            "year": "2018"
        }

        #Profile_2 etc etc
    }
}

Code

    with open('profileMulti.json', 'r', encoding='UTF-8') as json_data:
    config = json.load(json_data)

NameUrl = config["Url"]

myNote = config["My-Note"]

def checkoutNames(NameUrl, nameID):

#Request & other codes - Removed to recude the code
#......
#......
    headers = {
        'Referer': '',
        'Content-Type': ''
    }
    payload = {
        "shared": {
            "challenge": {
                "email": config["Email"],
                "PersonNumber": config["PersonNumber"],
                "postal_code": config["ZipCode"],
                "given_name": config["Name"],
                "Last_name": config["LastName"],
                "street_address": config["Address"],
                "postal_code": config["ZipCode"],
                "city": config["City"],
                "country": config["Country"],
                "email": config["Email"],
                "phone": config["Phone"],
            }

def checkoutNotes(NamesUrl, NamesPost):

#Request & other codes - Removed to recude the code
#......
#......

    headers = {
        'Accept': 'application/json, text/javascript, /; q=0.01',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Referer': NameUrl,
        'Connection': 'keep-alive'
    }
    payloadInfo = {
        "Information": {
            "Color": config["Color"],
            "house_number": config["houseNumber"],
            "year": config["Year"]
      }
    }    
def wipe():
    os.system('cls' if os.name == 'nt' else 'clear')

def main():
    time.sleep(1)

    FindName(myNote)

if _name_ == '_main_':
    try: {
        main()
    }
    except KeyboardInterrupt:
        wipe()

EDIT

I just wanted to print out what Yaroslav code gives for output

Yaroslav Surzhikov · Accepted Answer · 2017-09-23 14:23:56Z

1

You making list and expexcting it to work like dictionary. If order matters - use OrderedDict from collections. By the way, you could load you json using something like this:

from collections import OrderedDict
.... # your previous code
profiles_string = json.load(json_data, object_pairs_hook=OrderedDict)

Try with this code:

import json
from collections import OrderedDict
from multiprocessing.pool import ThreadPool
import threading
from time import sleep
import random

with open('profileMulti.json', 'r', encoding='UTF-8') as json_data:
  config = json.loads(json_data, object_pairs_hook=OrderedDict)


def stringify(key, next_layer):
  sleep(random.random()) # work emulation
  # `key` is profile name (profile_0, profile_1 etc)
  # `next_layer` is profile payload as OrderedDict:
  # {
  #    "Url": "Myasdwfaesite.se",
  #    "My-Note": "aasfase",
  #    "Email": "[email protected]",
  #    "PersonNumber": "5634543",
  #    "postal_code": "123445",
  #    "given_name": "Balling",
  #    "Last_name": "Calling",
  #    "street_address": "qwertr 123",
  #    "city": "London",
  #    "country": "UK",
  #    "phone": "65412331",
  #    "Color": "Blue",
  #    "house_number": "321",
  #    "year": "2018"
  #}

  # this print statement just for illustartion
  print(threading.current_thread().name, key)
  return { key: next_layer }  # still ordered

# how much workers should be started
profiles_number = len(config.get('Profiles', {}).items())

pool = ThreadPool(profiles_number)

# Launch a thread for each item and get() the results as each finishes
# `starmap_async` returns result object when all tasks are finished, so
# single get is called. Result object in thios case behave like list
results = pool.starmap_async(stringify, config.get('Profiles', {}).items()).get()

print('threaded results:')
for i in results:
    print(i, end='\n***\n')  # result separator when printed

edited Sep 23, 2017 at 14:23

answered Sep 23, 2017 at 12:25

Yaroslav Surzhikov

1,6081 gold badge13 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

WeInThis Over a year ago

Thank you @Yaroslav! However when I tried this. It did not work. What I get for print is [None, None]. What could cause that issue? Maybe because there isn't anything on print(item)? also if I want to print out etc. Url from the thread. Wouldnt in that case print(config['Url']) done it?

Yaroslav Surzhikov Over a year ago

@WeInThis, sorry, double return statement was fault. Try edited code, should be ok

WeInThis Over a year ago

Hey again! Ok so it worked. However I found another issue. So whats happening now is that it just loads into one thread and inside that thread there is the whole print of all the objects including all the profiles. I don't think it should act like that. However also. How would it be if I exemple want to grab out the URL. Etc Url = "Grab url from profile_0" and print(Url) ?

Yaroslav Surzhikov Over a year ago

@WeInThis i've added some comments to show what's happening in there

WeInThis Over a year ago

Oh you are right. It does now show the Thread-1 for profile_0 and Thread-2 for profile_1 which means it does work. You can see the picture I just added to see the output. Okey. So in that case that means it does give the right attributes. I have been trying to figure out now. Like I said before I do have ``` NameUrl = config["Url"] myNote = config["My-Note"] ``` inside my code and im just curious how I can in that case use those from the stringify to the attributes that iam wanting to give etc in payload ?

|

Collectives™ on Stack Overflow

Python Multithread/process combination of Json and Python?

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related