Type error: "a bytes-like object is required, not 'str'"

[faustus25]

Type error running Pysight.py under Python3:

Id": null, "observationTime": 1549856820}]}':
Unexpected error: %s (<class 'TypeError'>, TypeError("a bytes-like object is required, not 'str'",), <traceback object at 0x7ff411a33ac8>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "PySight.py", line 807, in <module>
    misp_process_isight_alert(result)
  File "PySight.py", line 778, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable

Line 778-791:

for i in a_result['message']:
        PySight_settings.logger.debug("  %s current element %s", len(a_result['message']), i)

        # USING THREADS to proceed with the resulting JSON
        if PySight_settings.use_threading:
            t = threading.Thread(target=isight_process_alert_content_element, args=(i,))
            t.start()
        else:
            # NO THREADING

            isight_process_alert_content_element(i)
            PySight_settings.logger.debug("Sleeping for %s seconds", PySight_settings.time_sleep)
            time.sleep(PySight_settings.time_sleep)
    end = timer()

Line 807:

misp_process_isight_alert(result)

Can you check if you encounter this issue?

[faustus25]

Some of the code comments point to this:

Line 142:

    new_data = a_query + '2.4' + 'application/json' + time_stamp
    # new_data=''
    # TODO: that is currently broken! TypeError: string argument without an encoding
    message = bytes(new_data, 'utf-8')
    secret = bytes(a_prv, 'utf-8')

[github-germ]

Hi, I have a local copy that I've mashed up and get_headers looks like this:

def get_headers(a_prv, a_pub, a_query):
    '''
    :param a_prv:
    :type a_prv:
    :param a_pub:
    :type a_pub:
    :param a_query:
    :type a_query:
    :return: headers for iSight search
    :rtype:
    '''
    time_stamp = email.utils.formatdate(localtime=True)

    new_data = a_query + '2.5' + 'application/json' + time_stamp
    # new_data=''
    # TODO: that is currently broken! TypeError: string argument without an enc\
oding
    message = bytes(new_data, 'utf-8')
    secret  = bytes(a_prv, 'utf-8')

    # hashed = hmac.new(bytearray(a_prv,'utf8'), new_data, hashlib.sha256)
    hashed = hmac.new(secret, message, hashlib.sha256)
    headers = {
        'Accept':         'application/json',
        'Accept-Version': '2.5',
        'X-Auth':         a_pub,
        'X-Auth-Hash':    hashed.hexdigest(),
        'Date':           time_stamp,
    }
    return headers

[faustus25]

Replacing the header function doesn't work either:

rId": null, "observationTime": 1549856820}]}':
Unexpected error: %s (<class 'TypeError'>, TypeError("a bytes-like object is required, not 'str'",), <traceback object at 0x7f54fefc9e48>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "PySight.py", line 837, in <module>
    misp_process_isight_alert(result)
  File "PySight.py", line 808, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable

Does your modified script work? and if so under python3.5?

I set up under virtual environment for python 3.5 but the encoding/decoding issue remains.

Can you share the entire script if it works for you?

[github-germ]

i think this is where that exception occurs: https://github.com/deralexxx/PySight2MISP/blob/master/PySight.py#L210

try changing this:

210         return_data_cleaned = r.data.replace('\n', '')

to this:

210         #return_data_cleaned = r.data.replace('\n', '')
211         return_data_cleaned = r.data

[faustus25]

No luck there either:

        #return_data_cleaned = r.data.replace('\n', '')
        return_data_cleaned = r.data
        # return_data_cleaned =

New error for object type:

Unexpected error: %s (<class 'FileNotFoundError'>, FileNotFoundError(2, 'No such file or directory'), <traceback object at 0x7fccf61eb348>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "PySight.py", line 838, in <module>
    misp_process_isight_alert(result)
  File "PySight.py", line 809, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable

[github-germ]

i made only that change to that github version, and it works for me. can you run the python debugger on this portion of the code and walk it through and see what's happening?

if that part of the code fails, it can return a False bool -- i suspect you are hitting one of those exception sections.

   208 	       PySight_settings.logger.debug("data %s: ", r.data)
   209
   210	       #commentout#return_data_cleaned = r.data.replace('\n', '')
   211         return_data_cleaned = r.data #addedin#
   212         # return_data_cleaned =
   213
   214 	       json_return_data_cleaned = json.loads(return_data_cleaned.decode\
('utf8'))
   215         PySight_settings.logger.debug(json_return_data_cleaned)
   216
   217	       # print json.dumps(theJson,sort_keys=True,indent = 4, separators\
 = (',', ': '))
   218         PySight_settings.logger.debug("Number of iocs: %s answer is: %s"\
, len(json_return_data_cleaned['message']),
   219                                       json_return_data_cleaned)
   220
   221         if not json_return_data_cleaned['success']:
...

[faustus25]

Bool object is the issue now as you mentioned.

Set the breakpoint for the block of code below:

 def isight_load_data(a_url, a_query, a_headers):
213         """
214
215         :param a_url:
216         :type a_url:
217         :param a_query:
218         :type a_query:
219         :param a_headers:
220         :type a_headers:
221         :return:
222         :rtype:
223         """
224         try:
225             PySight_settings.logger.debug("param headers: %s %s", a_headers, a_url)
226             proxy_request = ProxyManager(str(PySight_settings.proxy_adress))
227             url_to_load = PySight_settings.isight_url + a_query
228             PySight_settings.logger.debug(url_to_load)
229             try:
230
231                 r = proxy_request.request('GET', a_url + a_query, None, headers=a_headers)
232             except urllib.error.HTTPError as e:
233                 print(e.code)
234                 print(e.read())
235
236             PySight_settings.logger.debug("headers %s: ", proxy_request.headers)
237
238             PySight_settings.logger.debug("data %s: ", r.data)
239
240             #return_data_cleaned = r.data.replace('\n', '')
241             return_data_cleaned = r.data
242             # return_data_cleaned =
243
244
245             json_return_data_cleaned = json.loads(return_data_cleaned.decode('utf8'))
246             PySight_settings.logger.debug(json_return_data_cleaned)
247
248             # print json.dumps(theJson,sort_keys=True,indent = 4, separators = (',', ': '))
249             PySight_settings.logger.debug("Number of iocs: %s answer is: %s", len(json_return_data_cleaned['message']),
250                                           json_return_data_cleaned)
251
252             if not json_return_data_cleaned['success']:
253                 PySight_settings.logger.error("Error with iSight connection %s",
254                                               json_return_data_cleaned['message']['description'])
255                 PySight_settings.logger.error(json_return_data_cleaned)
256                 return False
257             else:
258  ->             import time
259                 timestring = time.strftime("%Y%m%d-%H%M%S")
260                 f = open("debug/" + timestring, 'w')
261                 f.write(json.dumps(json_return_data_cleaned, sort_keys=True, indent=6, separators=(',', ': ')))
262                 f.close()
263
264                 return json_return_data_cleaned
265         except:
266             print("Unexpected error: %s", sys.exc_info())
267             return False

Error below:

Unexpected error: %s (<class 'FileNotFoundError'>, FileNotFoundError(2, 'No such file or directory'), <traceback object at 0x7fabd9738d88>)
ERROR:PySight_settings:Something went wrong while downloading / processing the iSight Request
Traceback (most recent call last):
  File "/usr/lib/python3.5/pdb.py", line 1661, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib/python3.5/pdb.py", line 1542, in _runscript
    self.run(statement)
  File "/usr/lib/python3.5/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/user/python3-virtual-environments/PySight2MISP/PySight.py", line 838, in <module>
    misp_process_isight_alert(result)
  File "/home/user/python3-virtual-environments/PySight2MISP/PySight.py", line 809, in misp_process_isight_alert
    for i in a_result['message']:
TypeError: 'bool' object is not subscriptable
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program

[github-germ]

OK, there it is...

Unexpected error: %s (<class 'FileNotFoundError'>, FileNotFoundError(2, 'No such file or directory'), <traceback object at 0x7fabd9738d88>)

f = open("debug/" + timestring, 'w')

You need to mkdir debug in the directory where you are running this script.

[faustus25]

So it progressed further once the debug directory was added:

DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7ff5303bca48>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
DEBUG:PySight_settings:6 None
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted

installed libgcc next:

sudo apt-get install libgcc1-dbg

looked like it was connecting and downloading the i sight content then it complained about not being able to connect to my MISP instance, swapped the IP address of the instance with the alias version and reviewed the proxy details (same as in MISP instance):

DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:1 Inappropriate argument type.
ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6340598588>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f6340592fc8>)
ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6340591438>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f6340588508>)
ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6340591cc0>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f634058f588>)
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f63405b9b48>)
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f634059dfc8>)
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:1 Inappropriate argument type.
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f6340592fc8>)
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f636c0eb048>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:6 None
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:6 None
DEBUG:PySight_settings:2 (<class 'TypeError'>, TypeError('exceptions must derive from BaseException',), <traceback object at 0x7f6340588508>)
DEBUG:PySight_settings:3 <class 'TypeError'>
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:4 exceptions must derive from BaseException
  File "PySight.py", line 285, in isight_process_alert_content_element
    raise "no MISP instance found"
DEBUG:PySight_settings:6 None
DEBUG:PySight_settings:6 None
DEBUG:PySight_settings:6 None

MISP URL and proxy set as follows: (switched to misp_url=https;//127.0.0.1)

misp_url=https://misp.local.com/
misp_key=key
misp_verifycert=False

port=8080
host=http.proxy.com
protocoll=http
full=http://http.proxy.com:8080

So close to getting it to run smoothly!

[github-germ]

1. The code is not connecting to your MISP server as below. Are you sure you have the keys set correctly in config.cfg under the [MISP] section?

ERROR:PySight_settings:Unexpected error in MISP init: (<class 'pymisp.exceptions.PyMISPError'>, PyMISPError("Unable to connect to MISP (https://misp.local.com). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='misp.local.com', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6340598588>: Failed to establish a new connection: [Errno 110] Connection timed out',))",), <traceback object at 0x7f6340592fc8>)

2. And regarding this:

DEBUG:PySight_settings:1 Inappropriate argument type.

I would think this is is coming from one of the ConfigParser.get*() calls in Pysight_settings.py which is parsing the content in config.cfg -- can you check what you've put in config.cfg, and if not sure, can you paste that file in here?

[faustus25]

Config.cfg

[MISP]
misp_url=https://misp.localhost.com/
misp_key=key
misp_verifycert=False


[isight]
isight_url=https://api.isightpartners.com
isight_priv_key=key
isight_pub_key=key
last_hours=48


[proxy]
port=8080
host=http.proxy.com
protocoll=https
full=https://http.proxy.com:8080

[general]
use_threading=True
time_sleep=5

[github-germ]

OK... You need real values in your config.cfg.

1. I doubt your misp_url is what you hav in there. Set it to the URL of your MISP host.
2. You need to set misp_key to your user Authkey found on https://localhost/users/view/me (assuming your MISP is on localhost).
3. You ned to set your isight_priv_key and isight_pub_key to the keys supplied by your contact at iSIGHT.
4. Are you using a proxy? If so, I would think host and full are not correct.

[faustus25]

I have swapped those details naturally for the API keys for MISP and i Sight.

No quotes used in those fields,

[MISP]
misp_url=https://10.10.10.10/
misp_key=xxxrandomkeyxxx
misp_verifycert=False


[isight]
isight_url=https://api.isightpartners.com
isight_priv_key=xxxrandomkeyxxx
isight_pub_key=xxxrandomkeyxxx
last_hours=48


[proxy]
port=8080
host=http.proxy.com
protocoll=https
full=https://http.proxy.com:8080

[general]
use_threading=True
time_sleep=5

[github-germ]

HA, ok... Gotta see why you are getting those parser inappropriate args. Correct: no quotes in config.cfg. And then why is the connection to MISP failing? Can you write a quick Python script to try connecting to your MISP via PyMISP? If not, I can share some code for that.