Tag Archives: Records Management

Records Management

Better – Sensitive Information Types in Office 365

Sensitive Information Types in Office 365 allow you identify sensitive content that is held in Exchange and SharePoint Online and restrict its use.

You can leverage existing rules (e.g. Credit card numbers) or define your own.

The rules are applied as part of the search crawl. The content of a document or email is analysed and if, for example, a credit card number is found a property is set on the document.

Depending on the licence that you have for Office 365 you can then run searches to identify the offending content or apply policies that restrict its use.

Sensitive information types are not guaranteed to find every offending document but they are a great broad brush approach to information security and compliance.

If you are working with scanned images that have been OCR’d and converted to PDF+Text then there is a good chance that these will be identified.

Better – Kofax to HPE Content Manager Integration

HP provide an export connector (release script) for Kofax Capture to enable you to export scanned documents into HPE Content Manager (formerly HP Records Manager / Trim Context).

This has been around for a while and is used in most HPE Implementations in it’s most basic form.

You can find the reference information here.

Sometimes it’s good to look a better ways of doing things.

This article assumes that you are comfortable with the basic operation of the export connector but are looking at ways to improve, basically getting a more joined up capture system.

We are going to look at;

The Basic Experience
A Better Experience (For a lookup scenario)
Implementing that experience
- Configuring the Export Connector
- Writing a Lookup Function
- Calling the Lookup Function
- Fine Tuning

Ok so here goes.

Basic Experience

A lot of times the export connector is used in manual mode.

Every time a document is indexed the user sees the export connector (The panel underneath Trim Control Panel), clicks “Create Record”. They then get the Record Creation screen which they fill in using the mouse. If the names of your Kofax field and Trim fields align then they are defaulted (See Trim_Title in this example).

Better Experience

So a better experience is not to see the Export Connector at all. If the record can be created using just Kofax then manual indexing will be quicker and if you are using ICR / OCR / Recognition it can be potentially automated.

So we want something like below. We don’t really want to see that ugly record creation screen.

User just keys in the unique reference and the rest of the data should be pulled in from Records Manager, they can eyeball the image to make sure it matches, then the record should be automatically created.

Implementing

Ok so we know what we want but how to do.

Configuring the Export Connector

So to ensure that the export connector is working but not visible we set it to automatically invoke.

Writing a Lookup Function

So when the user keys in a unique reference we want to lookup information from Records Manager. So where is this going to come from? Well normally each record you scan is going into a folder and that folder will often carry all the meta data we need to visually check an image.

So our lookup is going to be to Records Manager which means a lookup function using the SDK.

This we can do using the .NET scripting built into Kofax.

' This sub looks up the folder and gets the details from the dedicated folder in Trim
Private Sub Lookup()

	' Connect to the default database on the Trim client
	Dim db As New Database
	db.Connect()

	' Create field definition for the user defined fields that we will be working with
	Dim fielddefLastName As New FieldDefinition(db, "Last Name")
	Dim fielddefFirstName As New FieldDefinition(db, "First names")
	Dim fielddefDOB As New FieldDefinition(db, "Date of Birth")
	Dim fielddefNI As New FieldDefinition(db, "National Insurance (NI) Number")


	'Set the Search criteria
	Dim titleCriteria As String
	Dim recordtypeCriteria As String

	titleCriteria = Employee_Number.IndexField.Value 

	recordtypeCriteria = FolderRecordType

	' Search object
	Dim records As New TrimMainObjectSearch(db, BaseObjectTypes.Record)

	' Build Search Criteria String
	Dim st As String
	st = String.Format("title:""{0}"" and type:[name:""{1}""]", titleCriteria, recordtypeCriteria)
	records.SetSearchString(st)


	For Each resultRecord As Record In records
		' Set the Kofax fields on the document to the values retrieved from the Trim folder for that person.
		Last_name.IndexField.Value = resultRecord.GetFieldValueAsString(fielddefLastName, StringDisplayType.Default, True)
		First_names.IndexField.Value = resultRecord.GetFieldValueAsString(fielddefFirstName, StringDisplayType.Default, True)
		Date_Of_Birth.IndexField.Value = resultRecord.GetFieldValueAsString(fielddefDOB, StringDisplayType.Default, True)
		NI_Number.IndexField.Value = resultRecord.GetFieldValueAsString(fielddefNI, StringDisplayType.Default, True)
		
		' Set the container for the scanned document to be the matching employee folder in TRIM.  I.e. put the record in the employees folder
		TRIM_Container.IndexField.Value = resultRecord.LongNumber
		
		' Set the title field of the scanned record 
		TRIM_Title.IndexField.Value = Employee_Number.IndexField.Value 

	Next

	db.Disconnect()

End Sub

' This sub looks up the folder and gets the details from the dedicated folder in Trim

Private Sub Lookup()

' Connect to the default database on the Trim client

Dim db As New Database

db.Connect()

' Create field definition for the user defined fields that we will be working with

Dim fielddefLastName As New FieldDefinition(db, "Last Name")

Dim fielddefFirstName As New FieldDefinition(db, "First names")

Dim fielddefDOB As New FieldDefinition(db, "Date of Birth")

Dim fielddefNI As New FieldDefinition(db, "National Insurance (NI) Number")

'Set the Search criteria

Dim titleCriteria As String

Dim recordtypeCriteria As String

titleCriteria = Employee_Number.IndexField.Value

recordtypeCriteria = FolderRecordType

' Search object

Dim records As New TrimMainObjectSearch(db, BaseObjectTypes.Record)

' Build Search Criteria String

Dim st As String

st = String.Format("title:""{0}"" and type:[name:""{1}""]", titleCriteria, recordtypeCriteria)

records.SetSearchString(st)

For Each resultRecord As Record In records

' Set the Kofax fields on the document to the values retrieved from the Trim folder for that person.

Last_name.IndexField.Value = resultRecord.GetFieldValueAsString(fielddefLastName, StringDisplayType.Default, True)

First_names.IndexField.Value = resultRecord.GetFieldValueAsString(fielddefFirstName, StringDisplayType.Default, True)

Date_Of_Birth.IndexField.Value = resultRecord.GetFieldValueAsString(fielddefDOB, StringDisplayType.Default, True)

NI_Number.IndexField.Value = resultRecord.GetFieldValueAsString(fielddefNI, StringDisplayType.Default, True)

' Set the container for the scanned document to be the matching employee folder in TRIM. I.e. put the record in the employees folder

TRIM_Container.IndexField.Value = resultRecord.LongNumber

' Set the title field of the scanned record

TRIM_Title.IndexField.Value = Employee_Number.IndexField.Value

db.Disconnect()

End Sub

What the script does is;

Connects to Trim
Finds the folder that matches our unique reference
Copies the meta from Trim back to Kofax so that the indexing user can see it
Ensures that when release the scanned document will be put in the correct folder (by setting the special Kofax field TRIM_Container
Ensures that the title is correct on the scanned document by setting the special Kofax field TRIM_Title

Calling the Lookup Function

The lookup function can be called as the post processing event on your unique lookup field.

Private Sub Unique_Number_FieldPostProcessing(ByVal sender As Object, ByVal e As Kofax.AscentCapture.NetScripting.PostFieldEventArgs) Handles Unique_Number.FieldPostProcessing


	Try
		' Get the details for the entered employee number
		Lookup()

	Catch ex As Exception
		MsgBox("Lookup Failed:" & vbCrLf & ex.Message)

	End Try


End Sub

Private Sub Unique_Number_FieldPostProcessing(ByVal sender As Object, ByVal e As Kofax.AscentCapture.NetScripting.PostFieldEventArgs) Handles Unique_Number.FieldPostProcessing

Try

' Get the details for the entered employee number

Lookup()

Catch ex As Exception

MsgBox("Lookup Failed:" & vbCrLf & ex.Message)

End Try

End Sub

Fine Tuning

Ok so you get the gist of one way of getting a better, more joined up capture experience for Kofax to Records Manager.

To get this ready for production you will also have to consider;

Kofax users moving backwards and forwards through batches and potentially changing values.
Setting a default destination container (This stops the manual record creation screen from randomly appearing in older export connector versions).

In Summary

The out of the box Kofax to Records Manager integration provides a good starting point.

With judicious use of scripting and configuration, however, you can make the two products really tightly integrated.

This can really reduce the time taken to scan documents especially with higher volume capture scenarios.

Simple Document and Meta Data Import to SharePoint

This article covers how to import documents to SharePoint. Creating a folder structure, meta data, importing files with content types and why it is important to do this with some consideration.

Scenario

We are going to assume that you have your documents on a file share / disk and that you have gathered your meta data in Excel, though your meta could quite easily be in a SQL or an Access database. We will further assume that the documents being uploaded are Staff Employment Records

Import Tool

We will use the free version of Import for SharePoint toolset to import the files.

Download Import for SharePoint (Free)

When you download the import tool you will have some Excel files, import configuration files, and screenshots of the content types which match this scenario. Using these will make the next steps much easier.

The import tool has a lot of functionality which is covered in the documentation

Bulk Folder Creation

We need a folder for each employee in our scenario.

We assume that you have created a document library, attached a custom “Staff Folder” content type and to that some site columns.

Using the import tool we can create the folders from our spreadsheet. The sheet is shown below.

From this the import tool will create a folder structure in your SharePoint library. Import meta data such as employee number is attached to each folder.

File Import

Now we can import our files into SharePoint.

We assume that you have created a document library, attached custom “Staff Document” and “Staff Disciplinary” content types and to those some site columns.

We can use the Excel spreadsheet as the import source.

Once the import has processed this the files will have been imported into the correct locations and with the correct meta data set against each one.

Why did we do this?

Ok so now we have a good structure to support common requirements.

Retention

How so let us assume that HR want to delete Staff folders 20 years after staff have left the business. We can add this retention policy onto the staff folder content type and for employees who have already left we have the date already set (See Bulk Folder Creation) .

Ok, so usually it’s a bit more involved that this but you get the point.

Search

Adding meta-data gives us a better chance of an item showing up in search results and in the instance of managed meta data will give us access to refiners on the search results page.

5000 Item Per Folder Limit

Ok so we know it’s not a good idea to have more than 5000 items in a folder. But doing our import as set out in this article you should be able to design a great structure that works inside this boundary even it your original folder structure did not.

So is this a packaged solution for Staff Records?

Well the reality is that the treatment of employment records will vary for each jurisdiction, can often complicated by different treatment for pension records, and the SharePoint implementation will change dependent upon whether you have a HR system and how that works.

That said it’s a great demo scenario and hopefully demonstrated some techniques will can be applied in all areas of your work.

Simple File Share Migration to SharePoint – Import

This article covers how and why to do a considered file share import to SharePoint. Creating a folder structure, meta data, importing files with content types and why it is important to do this.

Before you import your files you have hopefully prepared them and maybe have some Excel spreadsheets to import from.

No? Check out our article on preparing for import.

Scenario

So we are going to stick with the staff folder scenario prescribed in the above article. Your needs are likely to be different but you should be able to transfer the thinking and techniques demonstrated here.

Import Tool

We will use the Import for SharePoint toolset to import the files.

Download Import for SharePoint

Bulk Folder Creation

We need a folder for each employee in our scenario.

We assume that you have created a document library, attached a custom “Staff Folder” content type and to that some site columns.

Using the import tool we can create the folders from our spreadsheet. The sheet is shown below.

From this the import tool will create a folder structure in your SharePoint library. Import meta data such as employee number is attached to each folder.

File Import

Now we can import our files into SharePoint.

We assume that you have created a document library, attached custom “Staff Document” and “Staff Disciplinary” content types and to those some site columns.

We can use the Excel spreadsheet as the import source.

Once the import has processed this the files will have been imported into the correct locations and with the correct meta data set against each one.

Why did we do this?

Ok so now we have a good structure to support common requirements.

Retention

Ok, so usually it’s a bit more involved that this but you get the point.

Search

Adding meta-data gives us a better chance of an item showing up in search results and in the instance of managed meta data will give us access to refiners on the search results page.

5000 Item Per Folder Limit

So is this a packaged solution for Staff Records?

That said it’s a great demo scenario and hopefully demonstrated some techniques will can be applied in all areas of your work.

SharePoint WebHooks

Microsoft have recently announced the release of SharePoint Webhooks Developer Preview.

You can read the announcement here and further technical information here.

Webhook support in SharePoint provides an additional means to automate process in response to a change to or addition of a list item or document stored in SharePoint.

The mechanism allows you to register external application(s) with SharePoint online which will be called when an event occurs.

Historically many on premises SharePoint implementations used event receivers to run code in response to such events. Classic examples have included setting specific permissions on documents at point of creation, automatically creating folder structures, sending documents to record centres and so on. As event receivers are not permitted in SharePoint Online many organisations who have benefitted from using event receivers have been hesitant to migrate to the cloud. This announcement potentially eases migrations for such SharePoint users.

SharePoint – Modern Document Libraries – Limitations

Modern document libraries are now rolling out to all Office 365 commercial customers worldwide. You can learn more about how to use modern libraries in this article, “What is a document library?” or watch the video at, https://blogs.office.com/2016/06/07/modern-document-libraries-in-sharepoint/.

The new interface is attractive and easy to use. For the majority of SharePoint users it will be a very positive thing. It is not, however, without it’s limitations. These are of particular consideration for user’s heavily orientated towards document and records management.

So lets focus on the potential potholes for such customers.

Once multiple documents are selected you have limited capability to do anything nice like bulk check-in, you can only move, copy, delete.

The new quick edit function does not work well with multiple content types (it allows you to enter a data for a field which the document’s content type does not carry).

Linking through to editing a view is more complicated than before.

The way that the library interacts with the ribbon, masterpage and composed look is different so custom brands might be affected.

Sending a document to a record centre is not available because it is on the ribbon which is absent.

When you edit the properties of a document you CAN change the content type. When you edit the properties of a folder you CANNOT change the content type. Setting content types on folders is important in many records management / retention scenarios.

If you were using functionality change the view being used by virtue of the content type of the folder in which you were located then this is not fully observed by the new interface.

Now you can work around any of these limitations by clicking the “Return to classic SharePoint” in the bottom left hand corner but this does muddle the user experience.

You also have a certain level of control in regards to use of modern lists – See https://support.office.com/en-us/article/Differences-between-classic-and-new-experiences-for-lists-and-document-libraries-30e1aab0-a5cc-4363-b7f2-09e2ae07d4dc

In it’s current form (Summer 2016) the experience looks great for basic scenarios but for existing customers using SharePoint for document and records management should look before they leap.

Records Management – Sooner rather than later

The benefits of an improved Records Management system tend to come in the future and with retention periods for some records extending to decades that future can seen very distant indeed.

Sadly this tends to see records management projects placed on the proverbial back burner especially when budget it tight.

This, however, is probably not the wisest decision. Procrastination can be costly. Often when reviewing existing records and implementing a new solution deficiencies are found in the current meta data or record structure. In the most extreme cases a data cleansing exercise is needed and, of course, the problem is growing until addressed.

Take a case management scenario for example. When implementing compliant records management it is often the case that “Rejected” and “Accepted” cases needed to have vastly different retention to meet DP Act requirements. If case status has not been maintained against the system of record because this has not be considered then rework will be required.

For an alternative scenario consider employee records. When implementing compliant records management it is most certainly the case that some records such as pension details will have a very long retention whilst other such as a disciplinary action will be much shorter. If all the records have been stored simply in employee folders when again a restructure is likely required.

The truth of the matter is that until a RM project touches on the detail of requirements versus currently practices your organisation is perhaps unlikely to uncover these deviations between current and best practice.

So it might be time to resurrect that RM project after all.