Error during data conversion from Vensim (.vpmx) to Excel

[Condor323]

Dear reader,

My goal is to extract instances and the values of constants in a certain Vensim model and convert them to csv (Excel). When running the code i come across a type error which i cannot solve. I have already tried looking and making adjustments in the ema and ctypes libraries but to no avail. One adjustment from the original code was changing the extraction format for the vensim data from the vpm format in the code to vpmx.
I share the error and de full code under this post.

error

TypeError                               Traceback (most recent call last)
<ipython-input-8-6cf498b0d2f7> in <module>

13         typeNr = var_type
     14         print(var_type)
---> 15         varNames = get_varnames(filter=b'*', vartype=var_type) #
     16         for varName in varNames:                                        # per name, look in to their attributes
     17 

~\anaconda31\lib\site-packages\ema_workbench\connectors\vensimDLLwrapper.py in get_varnames(filter, vartype)
    384     filter = ctypes.c_char_p(filter)  # @ReservedAssignment
    385     vartype = ctypes.c_int(vartype)
--> 386     buf = ctypes.create_string_buffer("", 512)
    387     maxBuf = ctypes.c_int(512)
    388 

~\anaconda31\lib\ctypes\__init__.py in create_string_buffer(init, size)
     63         buf = buftype()
     64         return buf
---> 65     raise TypeError(init)
     66 
     67 def c_buffer(init, size=None):

TypeError: 

code

'''
Created on 24 June 2013

 

This script intends to sort the Vensim variables to a csv file.

 

@author: Willem L. Auping

'''


import csv

from ema_workbench.connectors.vensimDLLwrapper import get_varnames, get_varattrib

from ema_workbench.connectors.vensim import load_model


vensimRootName = r'C:\Users\coene\OneDrive\Documenten\EPA\Y2Q3\Capita selecta\project map'

vensimFileName = r'\20160323 Global Energy model - Climate policies'             # The name of the vensim model of interest

vensimExtension = r'.vpmx'

csvFileName = vensimFileName+'.csv'                                   # The name of the csv file


csvArray = ['Nr', 'Name', 'Equation', 'Unit', 'Comments', 'Type', 'Float', 'Int']   # The order of the elements in the array of every row

firstLine = ['Global Energy model data']                                         # The first lines of the csv file

secondLine = ['Model title',vensimFileName]

thirdLine = ['Time unit','Year']                                       # Write here the time unit of the model (or can it be found in the model?)

blank = ''

lineNumber = 1
 

attributeNames = ['Units', 'Comment', 'Equation', 'Causes', 'Uses', 'Initial causes',

                  'Active causes', 'Subscripts', 'Combination Subscripts', 'Minimum value',

                  'Maximum value', 'Range', 'Variable type', 'Main group']

attributes = range(len(attributeNames))

attributesInterest = [3, 1, 2, 12]

varTypeNames = ['All', 'Levels', 'Auxiliary', 'Data', 'Initial', 'Constant', 'Lookup',

                'Group', 'Subscript Ranges', 'Constraint', 'Test Input', 'Time Base',

                'Gaming']
varTypes = range(len(varTypeNames))


var_types = [1,2,3,4,5,6]

print('Vensim file: {}{}{}'.format(vensimRootName, vensimFileName, vensimExtension))

print('CSV file: {}{}'.format(vensimRootName, csvFileName))

print('Converting starts...')

load_model(vensimRootName + vensimFileName + vensimExtension)

with open (vensimRootName + csvFileName, 'w') as f:

    writer = csv.writer(f)
    writer.writerow(firstLine)                                      # The first lines are written
    writer.writerow(secondLine)
    writer.writerow(thirdLine)
    writer.writerow(blank)
    writer.writerow(csvArray)
    for var_type in var_types:                                            # Now get the variables per type
        type = varTypeNames[var_type]
        typeNr = var_type
        print(var_type)
        varNames = get_varnames(filter=b'*', vartype=var_type) # 
        for varName in varNames:                                        # per name, look in to their attributes

            csvArray[0]=lineNumber                                      # setup the line which needs to go to the csv

            csvArray[1]=varName

            csvArray[5]=varTypeNames[var_type]

            for attributeInterest in attributesInterest:                # put the attributes also in the line, in the right place

                attribute = get_varattrib(varName, attributeInterest)

                if attribute == []:

                    attribute = [blank]

                if attributeInterest == 1:                              # Unit

                    Unit = attribute[0]

                    csvArray[3] = Unit

                elif attributeInterest == 2:                            # Comment

                    csvArray[4] = attribute[0]

                elif attributeInterest == 3:                            # Equation

                    equation = attribute[0]

                    equation = equation.lstrip(varName)

                    equation = equation.replace(r'\\n','')

                    equation = equation.replace(r'\\t','')

                    equation = equation.replace(r'\n','')

                    equation = equation.replace(r'\t','')

                    equation = equation.replace(r'  ','')

                    equation = equation.lstrip('=')

                    equation = equation.lstrip(r' ')

                    csvArray[2] = equation

                if typeNr == 4:                                         # if type is Initial

                    csvArray[6] = 'x'                                   # The value for the 'Float' column is 'x'

                    csvArray[7] = blank                                 # The 'Int' column is empty

                elif typeNr == 5:                                       # if type is Constant

                    if varName[0:5] == 'Delay':

                        csvArray[6] = blank

                        csvArray[7] = ', integer=True'

                    elif varName[0:6] == 'Switch':

                        csvArray[6] = blank

                        csvArray[7] = ', integer=True'

                    else:

                        csvArray[6] = 'x'

                        csvArray[7] = blank

                else:

                    csvArray[6] = blank

                    csvArray[7] = blank   

                    
#            print csvArray

            writer.writerow(csvArray)

#            print varTypes

            lineNumber += 1

print('Converting ended.')

[quaquel]

you now give a long, and from the looks of it rather dated, script as example of what you try to do. This makes it hard for me to fully understand what you are trying to achieve. Can you isolate the problem and give a minimum working example of the error?

My hunch is that the error has something to do with bytes versus unicode strings in python. This is a major change that happened in between 2013 and now.

[Condor323]

main goal: i want to extract model data from Vensim into Excel. I am also only interested in the values of the constants in the model, i want to put each constant and its value(s) in excel so i can create an overview.

The error follows after executing line:
varNames = get_varnames(filter=b'*', vartype=var_type) #

Maybe not coincidental, this is the first snippet of code that actually tries to extract something from the Vensim data.
I hope this clears things up.

[quaquel]

My hunch is that line 386 in vensimDLLwrapper needs to be changed from buf = ctypes.create_string_buffer("", 512) to buf = ctypes.create_string_buffer(b"", 512). I don't have easy access to a windows machine at the moment to test this, however.

[Condor323]

Your hunch was right, changing buf = ctypes.create_string_buffer("", 512) to buf = ctypes.create_string_buffer(b"", 512) solves the issue.
I propose implementing this fix for the function get_varattrib( lines 321 and 328) and for the function get_varnames (lines 342, 386 and 390)

[Condor323]

Within the vensimDLLwrapper.py

[Condor323]

Also the l.strips in lines 335 and 397 must be changed from result = result.lstrip("'") to result = result.lstrip("b'")

[quaquel]

I think the entire dll wrapper needs to be checked and all strings that are not binary strings need to be changed.

[quaquel]

this has been fixed at some point in the meantime