Creating geoprocessing Toolboxes - Python script tool for automating the publishing of Revit model as Scene Layer

[KhaledAlhoz]

Introduction 

In day to day tasks, we often use ArcGIS Pro to carry out many processes and run many geoprocessing tools to manage our content and keep it up to date. For example, you could spend a few hours weekly on updating the information/data on your Web Server such as the content of your ArcGIS Online (organizational)-account. However, this is quite time-consuming to spend weekly on busy work. Python API is a powerful tool that helps you create your own geoprocessing tool that runs a list of executables. Even better, you can schedule the task to run the pipeline as frequently as you wish. This blog will show you how to convert a standalone Python script into a Python script toolbox and schedule it on ArcGIS Pro. Therefore, I will use a script created earlier which can be downloaded (together with the script-toolbox derived from it in this blog) from this GitHub repository. Also, if you are interested, this StoryMap explains and walks you through the steps of creating the script and automating the publishing of Revit Model to Scene Layer - the story map explains how to use Python API for automating the workflow as a standalone script. As a continuation, this blog will explain how to create a geoprocessing tool (script tool) that executes the same workflow using Python. 

The flowchart above shows the entire workflow or publishing a Revit model to scene layer. In the script, we have divided this workflow into two parts that we will be calling functions.

Function 1

def CreateBSLpackage(): 

• This function converts a Revit building model into a building scene layer package (.slpk) file in your local machine. This function executes the following pipeline:
  
  • Checking if the (.rvt) file at hand is modified since the last run (This is done by a helper function that can be turned on and off)
  • Creating geodatabase folder in a local folder
    
  • Importing Revit data into the geodatabase created as patches in a feature dataset
  • If "desired" make some changes on the patches e.g. delete a discipline or add attributes.
  • Making Building Layer from the feature dataset
  • Creating Building Scene Layer Package (.slpk) from the Building Layer
    

Function  2

def publishBSLfunction():

• This function takes the created building scene layer package (.slpk) and upload (or update an existing one) it in your content on ArcGIS Online server then publish (or replace an existing one) it as a scene layer (hosted).
  • Uploading (or updating) the created Building Scene Layer into ArcGIS Online
  • Finally, publishing (or replacing) the Scene Layer (Hosted); if the Scene Layer (Hosted) already exist, the replace function will create an Archive[name of the file][today date] version.
    

Table of Content

• Introduction
• Table of Content
• Body structure
• Walk through the steps of setting up the developing environment.
• Show an example of defining parameters from the script tool dialog box (user interface).
• Share the final script before and after to give an example on how to manipulate the input parameters...
• Setting up the environment for developing your own script geoprocessing tool.
• Example of defining parameters from the script tool dialog box
• Comparison of the script parameterization before and after:
• Recommendation for the future:

Body structure

In the remainder of this blog we will:

• Walk through the steps of setting up the developing environment.

• Show an example of defining parameters from the script tool dialog box (user interface).

• Share the final script before and after to give an example on how to manipulate the input parameters in the Python script  

Requirements for following the tutorial:

1. AcrGIS Pro
2. An ArcGIS organizational or developer account (if you're logged in ArcGIS Pro, this works automatically) 
3. A python script (from the GitHub repository you can download the script used here)
4. You're all set, let's start creating our customized geoprocessing tool!!

• Setting up the environment for developing your own script geoprocessing tool.

Before starting setting up the development environment, it might be useful to watch this awesome tutorial on how to create a script tool. It walks you through the same steps for a small script! 

1. In ArcGIS Pro create a new Toolbox 
   1. In the Catalog window, navigate to the folder or geodatabase where you want the toolbox to be created. 
   2. Right-click the folder or geodatabase and click New > Toolbox. Then, you can enter the name for your toolbox; Here, I named it CreateToolboxUsingPython
      

       

2. Right-click on the toolbox and click New > Script
   
   
   

3. The following dialog box pops up:

   1. Name - name of the tool
   2. Label - the label is the display name for the script tool (as shown in the Geoprocessing pane) and can contain spaces.

   3. Keep the Import script checkbox unchecked during the developing phase for testing purposes. After you finish the script and you are content with it, you can check the checkbox for embedding the script in the toolbox, help - Add a script tool. 

      
      
      
4. Next you click on parameters > the following dialog box appears:
   1. In this documentation you can find a detailed description of how to define each type of the parameters. Here you can design the interface parameters for user inputs. That is basically the arguments that your function(s) take as input.  
      
      
5. Now open the script you have at hand (possibly the one you have downloaded). In my script the parameters to be defined in the "__main__" space are as follows for CreateBSLpackage() function:

def CreateBSLpackage(workSpaceEnv = None, GDBfolder_name    = r"AutomationGDB.gdb",\
	out_FeatureDataset= r"Building_A"   , spatial_reference = r"RD New", \
	Rvt_directory     = None            , BSL_name          = r"BSLpackage.slpk",\
	nameOfBuildingL   = r"BuildL_A"     , includeDate       = False):
if __name__ == '__main__':
	run_CreateBSLpackage  = True
	"""required parameters and Optional parameters for the workflow
	this workflow is devided into two main parts (here functions)
	                   #################################
	                   ###Function 1 CreateBSLpackage()#
	                   #################################
	"""
	# Required
	########## 
	workSpaceEnv      =  r"C:\Users\alhoz\Desktop\AutomationTest" # path to a folder to createGDB
	Rvt_directory     =  r"C:\Users\alhoz\Desktop\AutomationTest\revitFiles\171025_BLOKA.rvt" # path to rvt file 
	BSL_name          =  r"BSLpackageBlockNew.slpk" # BSLpackage Name defaul is **BSLpackage.slpk**
	spatial_reference =  r"RD New" #default is "RD New"
	nameOfBuildingL   =  r"BuildL_Anew" #this will show on the ArcGIS online 
	# optional
	########## 
	out_FeatureDataset= r"Building_A"
	GDBfolder_name    = r"AutomationTESTnew.gdb" #default
	includeDate       = False ​​​

• Example of defining parameters from the script tool dialog box 
  

  Now the environment is set. In this part, we will show an example of deriving three parameters from the geoprocessing tool. For passing on the input of the user as an argument we will be using arcpy.GetParameterAsText() function from ArcPy package 

• Let's start with "run_CreateBSLpackage = True" argument; which is the first argument: 
  This is a boolean argument that takes the values True or False. That becomes the following:

  run_CreateBSLpackage = arcpy.GetParameterAsText(0)

• second parameter is workSpaceEnv; - a path to an empty folder on your local Machine;
  This becomes the following:

  workSpaceEnv = arcpy.GetParameterAsText(1)

• The third parameter Rvt_directory - path to Revit file
  This become the following, and so on for all the parameters that you wish to have as input from the user:

  Rvt_directory = arcpy.GetParameterAsText(2)

At this stage, the full script below takes as input for the three arguments from the wizard of the geoprocessing tool. In the next step, we will make sure that the user passes on the right input through the geoprocessing tool interface.

if __name__ == '__main__':
	run_CreateBSLpackage  = arcpy.GetParameterAsText(0)
	"""required parameters and Optional parameters for the workflow
	this workflow is devided into two main parts (here functions)
	                   #################################
	                   ###Function 1 CreateBSLpackage()#
	                   #################################
	"""
	# Required
	########## 
	workSpaceEnv      =  arcpy.GetParameterAsText(1)
	Rvt_directory     =  arcpy.GetParameterAsText(2) # path to rvt file 
	BSL_name          =  r"BSLpackageBlockNew.slpk" # BSLpackage Name defaul is **BSLpackage.slpk**
	spatial_reference =  r"RD New" #default is "RD New"
	nameOfBuildingL   =  r"BuildL_Anew" #this will show on the ArcGIS online 
	# optional
	########## 
	out_FeatureDataset= r"Building_A"
	GDBfolder_name    = r"AutomationTESTnew.gdb" #default
	includeDate       = False ​​​

• In dialog box from step 4 (setting up developing environment) you can add Label, Name and other properties for the three parameters at hand.
  For example, as mentioned before, Run_CreateBSLpackage is a boolean parameter. Therefore, we cheese Data Type: Boolean. Type: Input. The rest of the properties is pretty much straightforward. This snippet shows the properties of the three parameters we defined above. Fill them out and click ok.
  
  
  
• Click on the scripting tool in the catalog pane. Now the dialog box of the scripting tool should look something like this, and in the script the parameter takes its value from the geoprocessing tool interface. Note that some extra scripting is needed for getting the values in the correct datatype; for examples refer to the final script below.
• Now, define the inputs parameters through the wizard and click on  to test whether the workflow runs with no errors. If so, you continue with all parameters that should be taken as input from the user.

Once you've finished defining all parameters, and if you have worked on the same script; the tool should look similar to the snippet below. Now you can define the parameters and automate the process using the scheduler.  
Geoprocessing script-tool - Publishing Revit model to Scene Layer

Here are the final parameters' properties:

• Comparison of the script parameterization before and after:

  The comparison helps you learn how to manipulate the user input in the Python script. There are some data types that are a bit different in python, comparing with the input from the wizard, which need to be converted. 
  • Script before:
    

if __name__ == '__main__':

	run_CreateBSLpackage  = True
	run_publishBSLfunction= True
	checkDateOfRevitFile  = True # only works if you add (TimesLog.txt) file and the directory to it
	
	###########################################################################################################
	#directry to the text file in which the histoy log is stored and used for the date check of revit files####
	## see helper function:)
	directoryToTXTfile    = r"C:\Users\alhoz\Desktop\Automation\FinalPythonCodes\TimesLog.txt" 

	"""required parameters and Optional parameters for the workflow
	this workflow is devided into two main parts (here functions)

	                   #################################
	                   ###Function 1 CreateBSLpackage()#
	                   #################################
	"""

	# Required
	########## 
	workSpaceEnv      =  r"C:\Users\alhoz\Desktop\AutomationTest" # path to a folder to createGDB
	Rvt_directory     =  r"C:\Users\alhoz\Desktop\AutomationTest\revitFiles\171025_BLOKA.rvt" # path to rvt file 
	BSL_name          =  r"BSLpackageBlockNew.slpk" # BSLpackage Name defaul is **BSLpackage.slpk**
	spatial_reference =  r"RD New" #default is "RD New"
	nameOfBuildingL   =  r"BuildL_Anew" #this will show on the ArcGIS online 
	
	# optional
	########## 
	out_FeatureDataset= r"Building_A"
	GDBfolder_name    = r"AutomationTESTnew.gdb" #default
	includeDate       = False 
	####################################
	# check date if True and run the Function :CreateBSLpackage
	if checkDateOfRevitFile and run_CreateBSLpackage:
		run_CreateBSLpackage = checkDateFunction(Rvt_directory,directoryToTXTfile)
		if not run_CreateBSLpackage:
			run_publishBSLfunction = False
			print ("The last version of the Revit file is already uploaded")
	if run_CreateBSLpackage:
		CreateBSLpackage(workSpaceEnv, GDBfolder_name       , \
			       out_FeatureDataset, spatial_reference    , Rvt_directory, BSL_name,\
				   nameOfBuildingL   , includeDate )

	#####################################################################
	#####################################################################
	#####################################################################
	"""                ###################################
	                   ###Function 2 publishBSLfunction()#
	                   ###################################
	"""
	# Required
	##########
	itemID_BSLp             = "a3826c603e0d4f7d8c2a79ea35a88465" 
	itemID_Hosted           = "0af1fd92f95d46768a72b59845175bae"
	# overwrite parameter is importatnt to be set on **True** ##Only change if you know what you are doing##  
	dictOfPackageLayer      = {"overwrite" : True}
	DirectoryTo_SLPK        = None
	#### this parameter is automaticlly derived from the arguments of function 1 
	#### do not change it unless you know what you are doing
	if includeDate:
		now = datetime.now()
		day = now.strftime("%Y%m%d")
		BSL_name = BSL_name[:-5] +day + BSL_name[-5:] 
	if  DirectoryTo_SLPK is None:
		DirectoryTo_SLPK  = workSpaceEnv + "\\" + BSL_name

	if run_publishBSLfunction:
		publishBSLfunction(itemID_BSLp, itemID_Hosted, dictOfPackageLayer, DirectoryTo_SLPK)

• Script after

if __name__ == '__main__':


	run_CreateBSLpackage  = arcpy.GetParameterAsText(0)
	if run_CreateBSLpackage == "true":
		run_CreateBSLpackage = True
	else:
		run_CreateBSLpackage = False
	
	run_publishBSLfunction= arcpy.GetParameterAsText(9)
	if run_publishBSLfunction == "true":
		run_publishBSLfunction = True
	else:
		run_publishBSLfunction = False

	checkDateOfRevitFile  = arcpy.GetParameterAsText(13) # only works if you add (TimesLog.txt) file and the directory to it
	if checkDateOfRevitFile == "true":
		checkDateOfRevitFile = True
	else:
		checkDateOfRevitFile = False 
	
	###########################################################################################################
	#directry to the text file in which the histoy log is stored and used for the date check of revit files####
	## see helper function:)
	directoryToTXTfile    = arcpy.GetParameterAsText(14) 

	"""required parameters and Optional parameters for the workflow
	this workflow is devided into two main parts (here functions)

	                   #################################
	                   ###Function 1 CreateBSLpackage()#
	                   #################################
	"""

	# Required
	########## 
	workSpaceEnv      =  arcpy.GetParameterAsText(1) # path to a folder to createGDB
	Rvt_directory     =  arcpy.GetParameterAsText(2) # path to rvt file 
	BSL_name          =  arcpy.GetParameterAsText(3) + ".slpk" # BSLpackage Name defaul is **BSLpackage.slpk**
	spatial_reference =  arcpy.GetParameterAsText(4)  #default is "RD New"
	nameOfBuildingL   =  arcpy.GetParameterAsText(5) #this will show on the ArcGIS online 
	# optional
	########## 
	out_FeatureDataset= arcpy.GetParameterAsText(6)
	GDBfolder_name    = arcpy.GetParameterAsText(7) + ".gdb" #default
	includeDate       = arcpy.GetParameterAsText(8)
	if includeDate == "true":includeDate = True
	else:includeDate = False  
	####################################
	# check date if True and run the Function :CreateBSLpackage
	if checkDateOfRevitFile and run_CreateBSLpackage:
		run_CreateBSLpackage = checkDateFunction(Rvt_directory,directoryToTXTfile)
		if not run_CreateBSLpackage:
			run_publishBSLfunction = False
			arcpy.AddMessage ("The last version of the Revit file is already uploaded")

	if run_CreateBSLpackage:
		CreateBSLpackage(workSpaceEnv, GDBfolder_name       , \
			       out_FeatureDataset, spatial_reference    , Rvt_directory, BSL_name,\
				   nameOfBuildingL   , includeDate )


	#####################################################################
	#####################################################################
	#####################################################################
	"""                ###################################
	                   ###Function 2 publishBSLfunction()#
	                   ###################################
	"""
	# Required
	##########
	itemID_BSLp             = arcpy.GetParameterAsText(10)
	if itemID_BSLp  =="": itemID_BSLp  = None
	itemID_Hosted           = arcpy.GetParameterAsText(11) 
	if itemID_Hosted=="": itemID_Hosted= None

	# overwrite parameter is importatnt to be set on **True** ##Only change if you know what you are doing##  
	dictOfPackageLayer      = {"overwrite" : True}
	DirectoryTo_SLPK        = arcpy.GetParameterAsText(12)
	#### this parameter is automaticlly derived from the arguments of function 1 
	#### do not change it unless you know what you are doing
	if includeDate:
		now = datetime.now()
		day = now.strftime("%Y%m%d")
		BSL_name = BSL_name[:-5] +day + BSL_name[-5:] 
	if  DirectoryTo_SLPK == "":
		DirectoryTo_SLPK  = workSpaceEnv + "\\" + BSL_name

	if run_publishBSLfunction:
		publishBSLfunction(itemID_BSLp, itemID_Hosted, dictOfPackageLayer, DirectoryTo_SLPK)​

Let's take the same boolean input from example one:

run_CreateBSLpackage = arcpy.GetParameterAsText(0)
if run_CreateBSLpackage == "true":
    run_CreateBSLpackage = True
else:
    run_CreateBSLpackage = False

The wizard passes on the value in type string to the script, therefore, Python does not recognize Boolean (True of False) value. It simply read as a word true. There is a way around it, which is to simply change the value to Boolean, with if statement, as shown in the example above.   

If you check the rest of the input parameters, you will find examples of dealing with other types of input from the wizard. Script (before and after) and the toolbox BIMpublicationtoolbox can be found in the attachments, for if you wish to develop them or use them directly. Note that the same files, which will probably be improved in the future, can be downloaded from the GitHub repo. 

Recommendation for the future:

• In the current version, this reads only models in Revit (.rvt) format. In the future, once it is possible in ArcGIS Pro, this script can be developed to read different files format of BIM data such as (.ifc), (.dwg), and Civil 3D (.dwg), etc.
• Moreover, it can be interesting to research and test using Data Interoperability Tools to import data from different formats such as the ones just mentioned. Furthermore, you can examine combining patches (feature classes) from different data type sources in one geodatabase folder, before publishing.
• At the moment this script takes as input a Revit model from a folder on your local machine. As an enhancement, this script can be much improved if it can access BIM data directly from BIM cloud. Moreover, the new functionality - detecting modified Revit files in BIM 360 - can be utilized to check before downloading BIM models.
• This workflow is quite heavy on the machine, and it requires a powerful processor if the Revit model at hand is slightly big. Therefore, a further improvement, combined with the third recommendation, would be to schedule this script to run on the Online Notebook to make optimum use of the online server.

For any questions about this blog or any other questions, feel free to reach out.  

My name is Khaled Alhoz, an intern at Esri Nederland graduating from a master program in Geomatics at Tu Delft university in Delft, The Nederlands. My work focuses on GIS&BIM-integration related topics.

Contact info:
Khaled Alhoz
GIS-BIM intern at Esri Nederland 

LinkedIn
Emails:
kalhoz@esri.nl
k.alhoz.geo@gmail.com, 

My contribution are to fine on GitHub:
GitHub repositories

 

RvtToBuildingScnenLayer.zip