SHIFT

--- Sjoerd Hooft's InFormation Technology ---

User Tools

Site Tools


Sidebar

Recently Changed Pages:

View All Pages


View All Tags


LinkedIn




WIKI Disclaimer: As with most other things on the Internet, the content on this wiki is not supported. It was contributed by me and is published “as is”. It has worked for me, and might work for you.
Also note that any view or statement expressed anywhere on this site are strictly mine and not the opinions or views of my employer.


Pages with comments

View All Comments

cheatsheet

Table of Contents

Cheat Sheet

An overview of all sorts of commands and script usage.

PowerShell

PowerShell Help

Get-Help with -Parameter is a quick way to examine the data type a given parameter expects:
PS> Get-Help Get-ChildItem -Parameter Path
 
-Path <String[]>
    Specifies a path to one or more locations. Wildcards are permitted. The default location is the current directory (`.`).
 
    Required?                    false
    Position?                    0
    Default value                Current directory
    Accept pipeline input?       True (ByPropertyName, ByValue)
    Accept wildcard characters?  false

PowerShell History

View history of powershell commands and invoke them using “r #”
Get-History
<# Output #>
r 1


See all saved commands in vscode
code (Get-PSReadLineOption).HistorySavePath

PowerShell If

If statement
if (($alllogs -eq "True") -AND ($result)){
    Write-Host "Both statements are true"
}elseif (($buildreason -eq "IndividualCI") -or ($buildreason -eq "BatchedCI") -or ($buildreason -eq "PullRequest")){
    Write-Host "At least one of the statements is true"
}else {
    Write-Host "None of the statements is true"
}

PowerShell Count

Count number of strings / objects
# string
$een = "een"
$een.Count 
1
 
# Array
$twee = @("een","twee")
$twee.count
2


Length
# string
$een = "een"
$een.Length 
3
 
# Array
$twee = @("een","twee")
$twee.Length
2
$twee[0].Length
3


Measure-Object - Sometimes counting when there is only one item can fail, use measure-object instead.
$een = "een"
($een | Measure-Object).Count
1
$twee = @("een","twee")
($twee | Measure-Object).Count
2

PowerShell Dates

# Dates
$timestamp = Get-Date -format "yyyyMMdd-HH.mm" # 20211011-14.27
$readdate = Get-Date -format "d MMM yyyy" # 11 Oct 2021
$weekdate = Get-Date -uformat %V # Week number # 41
$monthdate = Get-Date -format "MMMM-yyyy" # May-2021


add or subtract days
[datetime]$endDate = (Get-Date).AddDays(+90)
[datetime]$endDate = (Get-Date).AddDays(-90)

Powershell Error Handling

Error handling explained


Try Catch Error Finally with LineNumber for troubleshooting

  1. function Get-ScriptLineNumber {
  2. return $MyInvocation.ScriptLineNumber }
  3. try{
  4. Write-Host "Do something that you want to catch the error of"
  5. }catch{
  6. # Catch the error
  7. Write-Host "Full Error: $_"
  8. # Or error out after all
  9. Throw "Failed on: $actionname; Status: Failed; Line: $(Get-ScriptLineNumber); Error: $($_.Exception.Message)"
  10. }finally {
  11. <#Do this after the try block regardless of whether an exception occurred or not#>
  12. # For example, clean up remote ps sessions
  13. }


Exception.options
try{
    # Do something that you want to catch the error of
    Get-ChildItem \\prox-hyperv -ErrorAction Stop
 
    # [System.Net.DNS]::GetHostByName("server99")
 
    # Connect-AzAccount
    # Get-AzResourceGroup -Name "exampleGroup" -ErrorAction Stop
}catch [System.Management.Automation.ItemNotFoundException] {
    Write-Host "Cannot find item."
    Write-Host -NoNewLine "Exception Message        : "
    Write-Host -Foreground Red -Background Black "$($_.Exception.Message)"
}catch{
    # Catch the error
    Write-Host -Foreground Red -Background Black "Error found. Please try again, or if the error persists, contact support for evaluation. Please provide the following diagnostics:"
    Write-Host -NoNewLine "Exception Message        : "
    Write-Host -Foreground Red -Background Black "$($_.Exception.Message)"
 
    # Exceptions can contain inner exceptions. This is often the case when the code you are calling catches an exception and throws a different exception. They will place the original exception inside the new exception.
    Write-Host -NoNewLine "Inner Exception Message  : "
    Write-Host -Foreground Red -Background Black "$($_.Exception.InnerException.Message)"
 
    # The FullyQualifiedErrorId is the .Message property of the exception object along with the the fully-qualified name of the class where the exception originated.
    Write-Host -NoNewLine "Fully Qualified Error Id : "
    Write-Host -Foreground Red -Background Black "$($_.FullyQualifiedErrorId)"
 
    # Output provides the error type you need to catch this specific error type
    Write-Host -NoNewLine "Exception type           : "
    Write-Host -Foreground Red -Background Black "$($_.Exception.GetType().fullname)"
 
    # Shows the  powershell command. Does not work for .Net exceptions
    Write-Host -NoNewLine "Command                  : "
    Write-Host -Foreground Red -Background Black "$($_.InvocationInfo.MyCommand)"
 
    # Position of the error in the scriptblock or file
    Write-Host -NoNewLine "Position                 : "
    Write-Host -Foreground Red -Background Black "At line: $($_.InvocationInfo.ScriptLineNumber) char:$($_.InvocationInfo.OffsetInLine)"
 
    # Shows the faulty line. Also works for .Net exceptions
    Write-Host -NoNewLine "Full line                : "
    Write-Host -Foreground Red -Background Black "$($_.InvocationInfo.Line)"
 
    # Shows the full error. Useful for troubleshooting. Provides output so use careful when returning output.
    Write-Host "Full Error:"
    Get-Error
 
    # Shows the full error for AZ cmdlets. Useful for troubleshooting. Provides output so use careful when returning output. Usually also works with non AZ cmdlets.
    Write-Host "Full Azure Error:"
    Resolve-AzError -Last
}finally {
    #Do this after the try block regardless of whether an exception occurred or not. For example, clean up remote ps sessions, clear variables or cleanup files.
}

PowerShell Input and Output

Enable logging of script
  1. ### Script Variables ###
  2. $scriptname = [System.IO.Path]::GetFilenameWithoutExtension($MyInvocation.MyCommand.Path.ToString())
  3. $scriptlocation = Split-Path $myinvocation.mycommand.path
  4. $date = (Get-Date).ToString('yyyyMMdd-HHmm')
  5.  
  6. ### Start transcript for full logging capabilities ###
  7. start-transcript -path "$scriptlocation\$scriptname-$date-logtranscript.txt"
  8.  
  9. ### Stop transcript
  10. Stop-transcript


Script output and Write-Host options
  1. # Inside a function use return or write-output
  2. return "value"
  3. Write-Output $value
  4. # Output an object without adding it to the function output
  5. Write-Host "color" -ForegroundColor Red -BackgroundColor green -noNewLine
  6. $object | Out-Host


Input password
$password = Read-Host "Enter password" -asSecureString


Use padright to fill a line with one symbol
Write-Host ''.PadRight(78, '=')
# Or with a variable
$wide = 32
Write-Host ''.PadRight($wide, '=')


Preference Variables

About Preference Variables

Verbose Preference
# Default
$VerbosePreference = 'SilentlyContinue'
# Override: -Verbose
 
# Verbose output
$VerbosePreference = 'Continue'
# Override: -Verbose:$false


Debug Preference
# Default
$DebugPreference = 'SilentlyContinue'
# Debug output
$DebugPreference = 'Continue'


Whatif Preference
# Default
$WhatIfPreference = $false
# Override: -WhatIf
 
# When WhatIf is enabled, the cmdlet reports the expected effect of the command, but doesn't execute the command.
$WhatIfPreference = $true
# Override: -WhatIf:$false

PowerShell Variables

Set system (environment) variable
$env:var="value"


Get all system (environment) variables, sorted by name
get-childitem -path env:* | Sort Name


Variables in function
Declare a variable with script scope to use it in a function
$script:orgUrl = "https://dev.azure.com/getshiftingcom"
$script:apiversion = "api-version=6.0"
function SetBuildTag {
  # Load Azure DevOps API settings
  AzureDevOpsAPISettings
  foreach ($tag in $tags){
      $tagurl = "$orgUrl/$env:System_Teamproject/_apis/build/builds/$env:BUILD_BUILDID/tags/$($tag)?$apiversion"
  }
}


set variable with a dash/hyphen
New-Variable -Name "MODULE-SiteName" -Value "Incomingapi"


set system variable with a dash/hyphen
[Environment]::SetEnvironmentVariable('MODULE-SiteName','Incomingapi')


Set system variables with a dash/hyphen
${env:test-app_user} = "svc-test-dev"
$user = ${env:test-app_user}

Remote PowerShell

Set variables
  1. $AdminCredentials = Get-Credential
  2. $remotePort = 5986
  3. $pso = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
  4. $Culture = "en-US"
  5. $pso.Culture = $Culture
  6. $pso.UICulture = $Culture
  7. $sessionParams = @{}
  8. $sessionParams.Add('UseSSL', $true)
  9. $sessionParams.Add('Port', $remotePort)
  10. $sessionParams.Add('Credential', $AdminCredentials)


Start session to restart computer
$session = New-PSSession -ComputerName webserverdmz.domain -EnableNetworkAccess -SessionOption $pso @sessionParams
Invoke-Command -Session $session -ScriptBlock {Restart-Computer}
Remove-PSSession $session

PowerShell Function Template

See here for guidelines.
function Use-AzureDevOpsApi{ #Use Get-Verb for a list of approved verbs
    <#
    .SYNOPSIS
        Short description
 
    .DESCRIPTION
        Long description
 
    .PARAMETER Pat
        Explain the parameter Pat. Repeat for additional parameters.
 
    .OUTPUTS
        Explain the powershell output, if any
 
    .EXAMPLE
        Pipeline example:
        - task: PowerShell@2
          displayName: "Use Default SystemAccessToken" 
          env:
            SYSTEM_ACCESSTOKEN: $(System.AccessToken) 
          inputs:
            pwsh: true
            targetType: 'inline'
            script: |
                # Load Azure DevOps API settings
                Use-AzureDevOpsApi
 
        Script example:
        Use-AzureDevOpsApi -Pat $Pat
 
    .LINK
        System access token: https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#systemaccesstoken
 
    .NOTES
        Author : Sjoerd Hooft
        Version: 2022-08-03 
 
    #>
    [CmdletBinding()]
    param (
        [Parameter()]
        [ValidateLength(52,52)] # See for more paramater validation https://poshcode.gitbook.io/powershell-practice-and-style/style-guide/function-structure 
        [ValidateSet("dev", "tst", "acc")]
        [string]
        $Pat
    )
 
    begin {
        Write-Host "##[section]Function: $($MyInvocation.MyCommand)"
        Write-Verbose "Input: $($PSBoundParameters | Out-String)"
 
        # Set Verbose switch based on PSBoundParameters
        $CmdLetOutput = @{Verbose = $($PSBoundParameters.Verbose)}
        # Collect information
    }
 
    process {
        # Do stuff
    }
 
    end {
        # Cleanup
    }
}

PowerShell Function Validates

  • Not null or empty: [ValidateNotNullOrEmpty()]
  • Latest or YYYY-MM: [ValidatePattern("latest|20\d{2}[-]\d{2}")]
  • Email: [ValidatePattern('(.+@getshifting\.com)$')]
  • Specific server name: [ValidatePattern("web(dev|tst|acc)dmz0[12]")]
  • Environment: [ValidateSet("dev", "tst", "acc", "prd")]
  • Number between: [ValidateRange(8,100)]
  • Length range: [ValidateLength(8,12)]
  • Length exactly: [ValidateLength(8,8)]

PowerShell Modules

Check Azure powershell module depending on PS version
  1. Write-Host "Check required PowerShell Modules `n"
  2. # The "new" PS module for Azure requires PowerShell 7
  3. if ($($PSVersionTable.PSVersion).Major -eq 7){
  4. if (Get-Module -ListAvailable -Name AZ) {
  5. Write-Host "PowerShell module Azure exists"
  6. } else {
  7. Write-Host "PowerShell module Azure does not exist. Start installation. "
  8. Install-Module -Name Az -Scope CurrentUser -Repository PSGallery -Force
  9. }
  10. }else{
  11. Throw "Powershell needs to be version 7"
  12. }


PowerShell DNS Module
if (Get-Module -ListAvailable -Name DnsServer) {
    Write-Host "PowerShell module DnsServer exists"
} else {
    Write-Host "Module DNSServer does not exist. Starting installation"
    Import-Module ServerManager
    Add-WindowsFeature -Name "RSAT-DNS-Server"
}


PowerShell AD Module
# Because ActiveDirectory is a 5.1 module you need to add -All 
if (Get-Module -ListAvailable -All -Name ActiveDirectory) {
    Write-Host "PowerShell module ActiveDirectory exists"
} else {
    Write-Host "Module ActiveDirectory does not exist. Starting installation"
    Import-Module ServerManager
    Add-WindowsFeature -Name "RSAT-AD-PowerShell" –IncludeAllSubFeature
}


PowerShell Pester module, remove shipped version and install latest version (Needs PS 7)
  1. if (Get-Module -ListAvailable -Name Pester) {
  2. # Check for version 3.4 and if available remove it
  3. Get-Module -ListAvailable -Name Pester
  4. if (((((Get-Module -ListAvailable -Name Pester).Version).Major) -eq 3) -AND ((((Get-Module -ListAvailable -Name Pester).Version).Minor) -eq 4)) {
  5. Write-Host "PowerShell Module Pester is version 3.4 which is shipped along with Win10/Windows 2016. Start removal:"
  6. $module = "C:\Program Files\WindowsPowerShell\Modules\Pester"
  7. takeown /F $module /A /R
  8. icacls $module /reset
  9. icacls $module /grant "*S-1-5-32-544:F" /inheritance:d /T
  10. Remove-Item -Path $module -Recurse -Force -Confirm:$false
  11. }
  12. # Verifieer of Pester al aanwezig met minimaal versie 5
  13. if ((((Get-Module -ListAvailable -Name Pester).Version).Major) -ne 5) {
  14. Write-Host "PowerShell module Pester is not up to date"
  15. Install-Module -Name Pester -Force -Scope CurrentUser
  16. }else{
  17. Write-Host "PowerShell Module Pester is available with minimal version of 5:"
  18. Get-Module -ListAvailable -Name Pester
  19. }
  20. } else {
  21. Write-Host "PowerShell module Pester does not exist. Start installation. "
  22. Install-Module -Name Pester -Force -Scope CurrentUser
  23. }


IIS Administration Module, install extra version, see the version and the available commandlets. Note that this needs to be done in PowerShell 5 Administrative Shell
Install-Module -Name IISAdministration -Scope AllUsers -Force #Installs latest version next to it (1.1.0.0)
Import-Module IISAdministration -RequiredVersion 1.1.0.0
$mod = Get-Module -Name IISAdministration
$mod.Version
$mod.ExportedCmdlets | Format-Table


Import PS Module
Import-Module Pester


Get commands within a module
Get-Command -module Pester


Get module for a command
Get-Command Get-Service


Remove / Unload a PowerShell module
Remove-Module Pester

PowerShell Requires

#Requires -Version 7.0
#Requires -PSEdition Core
# Requires for modules is slow because it also imports them
# Requires does not work for non-PS Core modules like DNS and ActiveDirectory
#Requires -Modules @{ ModuleName="Pester"; ModuleVersion="5.0" }
#Requires -Modules Az
#Requires -Modules AzureRmStorageTable

PowerShell ForEach

With custom PS Object and export to csv
  1. $myCol = @()
  2. foreach ($vm in (get-vmhost esxprd101.intranet | get-vm )){
  3. $VMInfo = "" | Select-Object VMName,VMHostName,NICCount
  4. $VMInfo.VMName = $vmview.Name
  5. $VMInfo.VMHostName = $vmview.Guest.HostName
  6. $VMInfo.NICCount = $vmview.Guest.Net.Count
  7. $myCol += $VMInfo
  8. }
  9. $myCol |Export-csv -NoTypeInformation $csvfile

PowerShell Comparison

Overview
# logical and comparison 
# -and, -or, -not , ! : Connect expressions 
# -eq, -ne : Equal, not equal 
# -gt / -lt, -ge / -le : Greater/less than, greater or equal / less or equal 
# -replace 
# -match / -notmatch : Regular expression match 
# -like / -notlike : wilcard matching 
# -contains / -notcontains : check for value in array: $array -contains $value
# -in / -notin : reverse syntax from contains: $value in $array 

PowerShell Location

Change directory (alias cd)
Set-Location c:\


Change directory and back
PS C:\Users\sjoer> Push-Location c:\
PS C:\> Pop-Location
PS C:\Users\sjoer>

Windows Management

Windows Service Management

Stop Windows Service
  1. # Stop the service while checking the state for x² seconds and kill it if that doesn't work
  2. Write-Host "Stopping Service: $servicename; Status: Initiating"
  3. $sleeper = 1
  4. while (((Get-Service -Name "$servicename").Status -ne "Stopped") -AND ($sleeper -lt 128)){
  5. if ($sleeper -eq 1){
  6. Stop-Service -Name "$servicename"
  7. } elseif ($sleeper -eq 32){
  8. Stop-Service -Name "$servicename" -Force -NoWait
  9. } elseif ($sleeper -eq 64){
  10. # Try to kill the process now
  11. $servicepid = (Get-CimInstance win32_service -Filter "Name = '$servicename'").ProcessId
  12. try{
  13. Stop-Process -id $servicepid -Force
  14. }catch{
  15. Throw "Stopping Service: $servicename; Stopping process: $servicepid; Status: Failed; Error: $($_.Exception.Message)"
  16. }
  17. }
  18. if ($alllogs -eq "True"){Write-Host "Stopping Service: $servicename; Sleeptime: $sleeper seconds"}
  19. Start-Sleep -Seconds $sleeper
  20. $sleeper = $sleeper * 2
  21. }


Restart all Windows Services
get-service -name grn* -ComputerName appprd02 | Restart-Service -Verbose


Start all stopped Windows Services
Get-Service -Name grn* -ComputerName appprd01,appprd02,appprd03,appacc01,appacc02,apptst01,apptst02,appdev01,appdev02 | Where-Object {$_.Status -eq 'Stopped'} | Start-Service -Verbose
Get-Service | where {($_.Status -eq 'Stopped') -and ($_.name -like 'Grn*') -and ($_.StartType -eq 'Automatic')} | Start-Service

Note: -ComputerName only works in PS 5

Install Windows Service depends on Powershell version
if ($($PSVersionTable.PSVersion).Major -eq 7){
    New-Service -Name $serviceName -BinaryPathName $binaryAppExecutable -StartupType "AutomaticDelayedStart" -Credential $appuserCredentials
}else {
    New-Service -Name $serviceName -BinaryPathName $binaryAppExecutable -StartupType "Automatic" -Credential $appuserCredentials
    sc.exe config $serviceName start= delayed-auto
}

Note: PS 5 does not understand the startuptype AutomaticDelayedStart

Delete Windows Service
sc.exe delete windows.service

Note: In cmd you can use sc, in powershell sc is an alias for set-content, so you need to use sc.exe

Windows IIS Management

Restart all App Pools
& $env:windir\system32\inetsrv\appcmd list apppools /state:Started /xml | & $env:windir\system32\inetsrv\appcmd recycle apppools /in

Note: This does not start stopped app pools

Restart all App Pools Remotely
$servers = 'web01','web02','web03'
Invoke-Command -ComputerName $servers { & $env:windir\system32\inetsrv\appcmd list apppools /state:Started /xml | & $env:windir\system32\inetsrv\appcmd recycle apppools /in }

Note:This does not start stopped app pools

Get the process for a IIS Site (always running under w3wp):
# Site and Appool have the same name:
$site = "customerapi"
processId = (Get-WmiObject -Class win32_process -filter "name='w3wp.exe'" | Where-Object { ($_.CommandLine).Split("`"")[1] -eq $site } ).ProcessId

Note: This one is also listed under Windows Process Management

Windows Certificates

View cert info
certutil -v -dump <path to cert>


Open Current User Certificate store:
certmgr.msc


Open Local Machine Certificate store:
certlm.msc


Get specific details from specific certificates from a list of servers and export to csv
$servers = @("server1","server2","server3")
Invoke-Command -ComputerName $servers {Get-ChildItem Cert:\LocalMachine\My | where-object {(($_.DnsNameList -like  "*rabobank*") -OR ($_.Subject -like "*rabobank*"))} |select-object @{name="hostname";expression={$(hostname)}},DnsNameList,NotAfter,Thumbprint,Subject,Issuer,FriendlyName} | Export-Csv certs.csv -NoTypeInformation

Windows Processes

Get all processes
$processes = Get-CimInstance -ClassName Win32_Process | Select-Object ProcessName,ProcessId,CommandLine,Path | sort processid


Kill a process
try{
    $processid = (Get-CimInstance win32_process -Filter "Name = 'mmc.exe'").ProcessId
    if ($null -ne $processid){
        Stop-Process -id $processid -Force
    }
}catch{
    Write-Host "Error killing mmc: $_"
}


Get the process for a IIS Site (always running under w3wp):
# Site and Appool have the same name:
$site = "customerapi"
processId = (Get-WmiObject -Class win32_process -filter "name='w3wp.exe'" | Where-Object { ($_.CommandLine).Split("`"")[1] -eq $site } ).ProcessId


Get the process and child processes:
$id = (Start-Process cmd.exe -PassThru).id
Get-WmiObject -Class Win32_Process -Filter "ParentProcessId = '$id' or ProcessId ='$id'" | Select-Object ParentProcessId,ProcessId,CommandLine


Get all process from a specific path
$dir = "appdir"
Write-Host "Alle processen in Path -like $dir"
Get-CimInstance win32_process -Property * | Where-Object {$_.path -like "*$dir*"}

Kill processes

Kill process and child processes
$id = (Start-Process cmd.exe -PassThru).id
Get-WmiObject -Class Win32_Process -Filter "ParentProcessId = '$id' or ProcessId ='$id'" | Select-Object ParentProcessId,ProcessId,CommandLine | ForEach-Object {Stop-process $_.processId}


Using taskkill
$id = (Start-Process cmd.exe -PassThru).id
# Killing the PID
taskkill /pid $id /t /f
# Or using the image name (executable name without path)
taskkill /im cmd.exe /t /f

Windows System Info

Collect uptime, patches, windows version and more
systeminfo 

System Variables

Set an environment variable until the end of the current command prompt session
set AWS_DEFAULT_REGION eu-west-1


Set an environment variable in both the current command prompt session and all command prompt sessions that you create after running the command
setx AWS_DEFAULT_REGION eu-west-1

Windows GPO

Group policy result to html
gpresult /h gpreport.html


Local Group Policy editor
gpedit.msc


GPO Management console:
gpmc.msc


Force GPO update
gpupdate /force

Windows Firewall

Allow incoming icmp rule to windows firewall
netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow

Windows Shutdown

Reboot in 1 second
shutdown -r -t 1


Shutdown in 0 seconds (force option implied)
shutdown -s -t 0

Active Directory

Active Directory Users

Count all users in all departments
Get-ADUser -SearchBase "OU=Users,DC=GetShifting,DC=local" -filter * -Properties name,department,enabled  | where {$_.enabled -eq "true"} | group department | select name, count


All enabled users that have never logged
Get-ADUser -Filter {(lastlogondate -notlike "*" -AND enabled -eq "True")} -SearchBase "OU=Users,DC=GetShifting,DC=local"


All enabled users that have not logged on for more than 90 days
$90Days = (get-date).adddays(-90)
Get-ADUser -Filter {(lastlogondate -le $90days -AND enabled -eq "True")} -SearchBase "OU=Users,DC=GetShifting,DC=local"


Get the most common reasons on a user for not being able to logon
Get-ADUser User123 -Properties * | Select-Object enabled,passwordexpired,lockedout

Active Directory Servers

All servers
Get-ADComputer -Filter {(operatingSystem -like "*windows*Server*")}

Note: Does not select Windows 2000 Servers.

Select servers on properties
Get-ADComputer -filter {(description -like "*financial*" -AND name -like "SQL*")} -Properties * | select name, description

Active Directory DNS

Get all AD DNS zones with all properties
Get-ChildItem "AD:CN=MicrosoftDNS,DC=DomainDNSZones,DC=getshifting,DC=local" | Select-Object *


Get all records in a DNS zone with all properties
Get-ChildItem "AD:DC=getshifting.local,CN=MicrosoftDNS,DC=DomainDNSZones,DC=getshifting,DC=local" | Select-Object *


Get creation and modify date of AD DNS records
Get-ChildItem "AD:DC=getshifting.local,CN=MicrosoftDNS,DC=DomainDNSZones,DC=getshifting,DC=local" | Where-Object {$_.name -like "build*"} | Get-ADObject -Properties Created,Modified | Select-Object Name,Created,Modified

Networking

Reverse loopup with ping
ping -a 10.10.10.10


NSlookup with default nameserver
nslookup servername


NSlookup with non-default nameserver ns01
nslookup servername ns01


Remote desktop console
mstsc.exe /v:servername /admin 


Test connection
Test-NetConnection -InformationLevel "Detailed" -Computername $host

Windows TCP Dump

Note: This needs (portable) tcpdump, the created file can be opened by (portable) wireshark for analysis. See here for more info on tcpdump.

dump traffic with host 10.10.10.10 on interface 1 to file
\tcpdump.exe -i 1 -vv host 10.10.10.10 -w tcpdumpfile.pcap 


dump traffic with host 10.10.10.10 on interface 1 to file with a readable timestamp format
\tcpdump.exe -i 1 -vv host 10.10.10.10 -tttt -nr tcpdumpfile.pcap 

Remote IPConfig

Renew Ip config remotely without losing access
ipconfig /release && ipconfig /renew

Azure DevOps

Output in Azure DevOps

Logging output
# Logging output in azure devops
##[group]Beginning of a group - white 
##[warning]Warning message - orange 
##[error]Error message - red 
##[section]Start of a section - green 
##[debug]Debug text - purple 
##[command]Command-line being run - blue 
##[endgroup] - green 
Write-Host "`n##[section]Function: $($MyInvocation.MyCommand).`n"


Log error or warning
Write-Host "##vso[task.logissue type=error]Something went very wrong."
Write-Host "##vso[task.logissue type=warning]Found something that could be a problem."


Generate extra output if debug mode is set
if ($env:SYSTEM_DEBUG -eq "True"){$headers | Out-Host}

Variables in Azure DevOps

Pipeline variable in yaml - without PowerShell
variables:
  - name: BuildConfiguration
    value: 'Release'
- task: DotNetCoreCLI@2
  displayName: "Build Solutions"
  inputs:
    command: 'build'
    projects: '**/*.sln'
    arguments: '--no-restore --configuration $(BuildConfiguration)'


Pipeline parameter in yaml
if ( "${{ parameters.alllogs }}" -eq "True" ){write-Host "Parameter from pipeline"}


System variable in yaml
Variables can be used like system environment variables

if (Test-Path "$env:Build_SourcesDirectory\$env:Build_Repository_Name\appsettings.Development.json"){Write-Host "System / Environment Variable"}
$sites = get-childitem -path env:* | Where-Object {$_.name -like "*-SiteName"} | Select-Object -ExpandProperty value # Added through Variable Group


Variables can also be used like pipeline variables

get-childitem $(Pipeline.Workspace) 
$dbserver = "$(serverDb)" + "." + "$(dnsZone)" # from Azure App Configuration


Variables in PS scriptfile
$adserver = $env:adServer + "." + $env:domain # from Azure App Configuration
$projects = Get-ChildItem -Path "$env:Build_SourcesDirectory\$env:Build_Repository_Name\deploy\" -Directory | Select-Object Name

Starter Azure Pipeline

name: $(Build.DefinitionName)-$(Build.BuildId)
appendCommitMessageToRunName: false

variables:
  buildConfiguration: 'Release'
  deploymentSuffix: '$(Build.DefinitionName)-$(Build.BuildId).$(System.StageAttempt).$(System.JobAttempt)'

parameters:
  - name: alllogs
    displayName: "Enable all logging"
    type: boolean
    default: true
 
# https://learn.microsoft.com/en-us/azure/devops/pipelines/process/scheduled-triggers?view=azure-devops&tabs=yaml
schedules:
- cron: "05 23 * * *" # 23.05h
  displayName: Nightly build on master
  branches:
    include:
    - master
  always: true

pool:
  name: ReleasePool # Self hosted
  vmImage: ubuntu-latest # MS Hosted
  vmImage: windows-latest # MS Hosted

trigger:
  branches:
    include:
      - release/*
      - master
trigger: none

resources:
  repositories:
  - repository: templates
    type: git
    name: DevOps/templates
  - repository: self
  pipelines:
  - pipeline: TA
    source: TestAutomation # Pipeline name that produces the artifact (must be unique)

stages:
- stage: build
  displayName: "Stage: Build"
  pool:
    name: BuildPoolDotNet

  jobs:
  - job: build
    displayName: "Job: Build & Test"
    steps:
    - checkout: self
      clean: true # If true, run git clean -ffdx followed by git reset --hard HEAD before fetching.
    - checkout: templatses
      clean: true

    - ${{ if not(eq(variables['Build.Reason'], 'PullRequest')) }}:
      - template: /pipelines/log-information.yml@operations
        parameters:
          environment: "environment: build"
          alllogs: ${{ parameters.alllogs }}
 
# Stage with template that has steps (see deploy.yml)
- stage: release
  displayName: "Stage: release"
  condition: and(succeeded(), ne(variables['Build.SourceBranchName'], 'master'))

  variables:
    environment: 'tst'

  jobs:
  - deployment: Deploy
    workspace:
      clean: all
    environment: ${{ variables.environment }}

  - template: /pipelines/deploy.yml@templates
    parameters:
      environment: ${{ variables.environment }}
      alllogs: ${{ parameters.alllogs }}
 
# Stage with template as step (see prod.yml)
- stage: production
  displayName: "Stage: prod"
  dependsOn: release

  variables:
    tag: 'Production'

  jobs:
  - job: prd
    displayName: "production"
    condition: and(succeeded(), eq( ${{ parameters.deployProduction }}, true))
    variables:
      environment: 'prd'

    steps:
    - template: /deploy/production.yml
      parameters:
        environment: ${{ variables.environment }}
        alllogs: ${{ parameters.alllogs }}

Template deploy.yml

parameters:
  environment: ""
  alllogs: ""
  dependsOn1: []
  dependsOn2: []
  dependsOn3: []
  dependsOn4: []

jobs: 
- job: deploy 
  displayName: "Deploy steps"
  dependsOn: 
    - ${{ parameters.dependsOn1 }}
    - ${{ parameters.dependsOn2 }}
    - ${{ parameters.dependsOn3 }}
    - ${{ parameters.dependsOn4 }}
  workspace:
    clean: all
  steps:

  - checkout: operations
    clean: true # If true, run git clean -ffdx followed by git reset --hard HEAD before fetching.
 
  #- ${{ if eq(parameters.alllogs, true) }}:
  - template: /pipelines/log-information.yml@operations
    parameters:
      environment: ${{ parameters.environment }}
      alllogs: ${{ parameters.alllogs }}
      input1: "adGroups: ${{ parameters.adGroups }}"
  
  - task: PowerShell@2
    displayName: "PowerShell task"
    inputs:
      pwsh: true
      targetType: 'inline'
      script: |
        Write-Host "`n##[section]PowerShell task `n"

Template prod.yml - Most Common Tasks

parameters:
  environment: ""
  alllogs: ""
  spokeEntity: ""

steps:
- task: PowerShell@2
  displayName: "Powershell Core task"
  inputs:
    pwsh: true
    targetType: 'inline'
    script: |
      Write-Host "`n##[section]Start PS Core task`n"
      Get-ChildItem -path env:* | Sort Name

- task: Bash@3
  displayName: "Bash task"
  inputs:
    targetType: 'inline'
    failOnStderr: true
    script: |
      echo '##[Section]Start Bash task'
      printenv | sort

- task: AzureCLI@2
  displayName: "Azure CLI task"
  condition: and(succeeded(), or(eq(variables.runAzureTask, true), eq(variables.runAzureAll, true)))
  inputs:
    azureSubscription: $(azureServiceConnection)
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      set -e
      echo '##[Section]Start Azure CLI task'
      az account set --subscription $(subId)

- task: AzurePowerShell@5
  displayName: "Azure Powershell task"
  inputs:
    azureSubscription: $(azureServiceConnection)
    azurePowerShellVersion: 'LatestVersion'
    pwsh: true
    ScriptType: 'InlineScript'
    Inline: |
      if ( "${{ parameters.alllogs }}" -eq "True" ){Write-Host "`n##[section]Start Azure Powershell task`n"}
      # Set Azure Context for environment subscription
      Set-AzContext -Name ${{ parameters.environment }} -Subscription $env:azureSubscriptionName

Docker Containers

Show running container
docker ps
# or
docker container ls


Show all containers
docker ps -a


docker stop/start/restart containers
# Stop
docker stop container_id
# default grace time = 10
docker stop --time=5 container_id 
# Start
docker start container_id
# Restart
# Restart apache container by name
docker container restart httpd_web


Get docker container info
docker inspect container_id


Get docker container compose info
docker inspect container_id | grep com.docker.compose


Copy a file from or to a container
docker cp tool_db:/var/lib/postgresql/data/pgdata/postgresql.conf ./postgresql.conf

Logging and Monitoring

Get logging from docker container
docker logs --tail 50 --follow --timestamps httpd_web


grep in logging
docker logs <container_name> 2>&1 | grep <string>

Note that docker logs to stderr do you need to redirect this to stdout

Get container stats
docker top <container_name>


Get docker stats
docker stats

Work Inside a Container

Work in a container
docker exec -it container_id /bin/bash


As a different user
docker exec -u postgres -it container_id /bin/bash

Docker Compose

Docker compose files are used to start containers. By default, a docker-compose.yml or a docker-compose.yaml file is used.


Start and stop containers from docker-compose file

docker compose up -d
docker compose down

Note the -d switch to start the containers in the background.

See logging
# the logs from each of the services interleaved into a single stream
docker compose logs -f
# the logs just for app
docker compose logs -f app

Docker Build

Docker build, run and show logs of a custom app called privacy
# Build an image called privacyimage
docker build -t privacyimage .
# Run a container called privacycontainer from the image
docker run -d --name privacycontainer privacyimage
# Show the logs from the container
docker logs privacycontainer


# Show all steps in plain output and redo every step (don't use cached steps)
docker build --progress=plain --no-cache  -t consoleappdbtestimage .


Save an image to disk and import somewhere else
# Create a tar file from the consoleappimage
docker save -o ./consoleappimage.tar consoleappimage
# import the image
docker load -i /consoleappimage.tar

Dockerfile

A Dockerfile is used to create images:

Default .net app dockerfile:

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /App
 
# Copy everything
COPY . ./
# Restore as distinct layers
RUN dotnet restore
# Build and publish a release
RUN dotnet publish -c Release -o out
 
# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "DockerConsoleApp.dll"]

Docker Pipeline Task

  - task: Docker@2
    displayName: Build and push an image to container registry
    inputs:
      command: buildAndPush
      repository: $(repository)
      dockerfile: $(dockerFilePath)
      containerRegistry: "containerRegistry"
      tags: |
        $(imageName)
        $(Build.BuildId)
        latest

Bash

Searching

grep file without empty lines and comments
grep -v '^\s*$\|^\s*\#' postgresql.conf


find file in current and subdirectories
sudo find . -name main.cf

Crontab

display all crontabs for all users as root sudo su -
for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done

File Management

Compare two files
diff file file2


Compare two files next to each other, only differences
diff -y --suppress-common-lines file1 file2

Linux

OS

See release
cat /etc/os-release

Vi

  • Turn off syntax :syntax off
  • Delete all below lines, including current: dG
  • Past text from windows clipboard through putty: SHIFT + INS

Certificate Management

Create csr with new private key (preferred)
openssl req -nodes -newkey rsa:2048 -keyout *.shiftwiki.nl.key -out *.shiftwiki.nl.key.csr


Create csr with existing key
openssl req -new -key /data/keys/httpd-wild.shiftwiki.nl/oeserver.pem  -out *.shiftwiki.nl.key.csr


View certificate
openssl x509 -text -noout -in certificate.crt


View csr

openssl req -text -in request.csr

Logfile Management

Truncate a logfile that has grown too big

sudo truncate -s 0 odoo.log

Disks

See link between device and filesystem for AWS EBS disks. From the instance, go to Storage tab and see the device name, then use these commands:

[sjoerd@server ~]$  df -hT /dev/sdf
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme1n1   xfs    20G   14G  6.8G  67% /var/lib/docker
[sjoerd@server ~]$  df -hT /dev/sdg
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme2n1   xfs    20G   15G  5.8G  72% /data

Azure

Azure PowerShell

NOTE: Module AZ only works in PS 7


TLS version
# Set TLS version to use TLS 1.2 
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Login to Azure

Login and context
Connect-AzAccount
Get-AzContext -ListAvailable
Set-AzContext -Name $dtapEnvironment -Subscription $azureSubscriptionName


Get TenantId
$tenantId = (Get-AzContext).Tenant.Id

KeyVault

Get secret from KeyVault
$adadminpass = (Get-AzKeyVaultSecret -VaultName kv-operations-global -Name "adAdminPass").SecretValue


Set secret to KeyVault
$userpasswordsecure = ConvertTo-SecureString -String $userpassword -AsPlainText -Force
Set-AzKeyVaultSecret -VaultName kv-operations-global -Name $username -SecretValue $userpasswordsecure

Resourcegroup Creation Time

((Invoke-AzRestMethod -Path "/subscriptions/$subId/resourcegroups?api-version=2020-06-01&`$expand=createdTime" -Method GET).Content | ConvertFrom-Json).value | Select-Object name,createdTime

Azure CLI

Login to Azure

az login
az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
az account show

Get Objects

# Check if a resourcegroup exists and set a variable
RgExists=$(az group exists -n $(rgName))
echo "$(rgName) exists: $RgExists"
 
# Get firewall Id
firewallId=$(az network firewall show --name $(azureFirewallName) --resource-group $(rgName) --query id --output tsv)
 
# Get Log Analytics Workspace Id
workspaceId=$(az monitor log-analytics workspace show --workspace-name $(lawName) --resource-group $(rgName) --subscription $(subId) --query id -o tsv)
 
# Get Virtual Network Gateway (VPN) Id
vngId=$(az network vnet-gateway show --name $(virtualNetworkGatewayName) --resource-group $(rgName) --query id --output tsv)
 
# Get Keyvault Id
keyvaultId=$(az keyvault show --name $(keyVaultName) --resource-group $(rgName) --query id --output tsv)

Tags

resourceGroup=$(az group show -n poc-$(team)-$(pocnaam) --query id --output tsv)
echo 'The resource group has the following tags'
az tag list --resource-id $resourceGroup
echo 'add a tag'
az tag update --resource-id $resourceGroup --operation Merge --tags DateOne="$(firstdate)" DateTwo="$(seconddate)"

Azure Resource Graph Explorer Query

# check if a resource has a specific tag set
tag=$(az graph query -q "resourcecontainers | where type == 'microsoft.resources/subscriptions/resourcegroups' and name == 'poc-$(team)-$(pocnaam)' | where tags contains 'startdate'" | jq '.data[].name')
 
echo "Tag date: $tag"
 
# do something if the tag is empty
if [ -z "$tag" ];
then
  echo "do something, the tag is not set"
else
  echo "do something else, the tag is set"
fi
 
# do something if the tag is not empty
if [ -n "$bedrijf" ];
then
  echo "do something, the tag is set"
else
  echo "do something else, the tag is not set"
fi

BICEP deployment

# deploy resourcegroup in a subscription
az deployment sub create \
  --location $(location) \
  --name "hub-$(deploymentSuffix)" \
  --template-file ./modules/resourcegroup.bicep \
  --parameters ./hub/hub-rg-parameterfile.jsonc \
  --parameters location=$(location)
 
# deploy resources in resourcegroup
az deployment group create \
  --resource-group $(rgName) \
  --name "hub-vnet-$(deploymentSuffix)" \
  --template-file ./hub/hub-vnet.bicep \
  --parameters deploymentSuffix=$(deploymentSuffix) \
  --parameters subId=$(subId) \
  --parameters vnetName=$(vnetName)

Kusto

These are just a few examples, see Kusto Queries in Azure for more.

Verify incoming logs per table
Perf
| where TimeGenerated > ago(10m)
| order by TimeGenerated desc
 
Event
| where TimeGenerated > ago(10m)
| order by TimeGenerated desc
 
RabbitMQ_CL
| where TimeGenerated > ago(10m)
| order by TimeGenerated desc
 
W3CIISLog
| where TimeGenerated > ago(10m)
| order by TimeGenerated desc


Log errors every 10 minutes in a stacked graph
APP_LOG
| where RawData contains "error"
| summarize count() by Computer, bin(TimeGenerated, 10m)
| order by TimeGenerated
| render areachart


Week overview of errors per service account in a column graph
APP_LOG
| where LogType_CF == '[error]'
| summarize count() by UserName_CF, Computer, bin(TimeGenerated, 1d)
| order by TimeGenerated, UserName_CF
| render columnchart


SQL timeouts in a column graph
exceptions
| where outerMessage has "timeout" and outerType == "Microsoft.Data.SqlClient.SqlException"
| summarize count() by bin(timestamp, 1d), cloud_RoleName
| order by timestamp
| render columnchart


All exceptions per 30 seconds per application
exceptions
| where outerMessage contains "timeout"
| where cloud_RoleName != ""
| summarize count() by bin(timestamp, 30s), cloud_RoleName
| order by timestamp


All high severity level exceptions of a specific type
exceptions
| where severityLevel > 2 and type == 'MassTransit.RequestTimeoutException'
| summarize count() by cloud_RoleName, bin(timestamp, 1d)
| render columnchart

Azure Resource Graph Explorer

Get all resources, type, location, subName and rgName
resources
| join kind=inner (
    resourcecontainers
    | where type == 'microsoft.resources/subscriptions'
    | project subscriptionId, subscriptionName = name)
    on subscriptionId
| project name, type, location, subscriptionName, resourceGroup
| order by type asc


Get all application insights resources without a workspace
resources
| where type == "microsoft.insights/components" 
| where properties['WorkspaceResourceId'] == ""
| join kind=inner (
    resourcecontainers
    | where type == 'microsoft.resources/subscriptions'
    | project subscriptionId, subscriptionName = name)
    on subscriptionId
| project name, type, subscriptionName, resourceGroup
| order by subscriptionName asc


Get all storage accounts with an old tls version
resources
| where type == "microsoft.storage/storageaccounts"
| where properties['minimumTlsVersion'] != "TLS1_2"
| project name, resourceGroup, properties.minimumTlsVersion

Azure DevOps Audit Stream

yaml and classic release that were not successful
AzureDevOpsAuditing
| where OperationName == "Pipelines.DeploymentJobCompleted" or OperationName == "Release.DeploymentCompleted"
| where Data.DeploymentResult != "Succeeded"
| extend deployment = strcat(Data.PipelineName, "-", Data.StageName)
| project TimeGenerated, deployment
| summarize count() by bin(TimeGenerated, 10m), deployment
| render columnchart


yaml releases to production that were successful
AzureDevOpsAuditing
| where OperationName == "Pipelines.DeploymentJobCompleted" and Data.DeploymentResult == "Succeeded" and Data.StageName contains "Production"
| extend deployment = strcat(Data.PipelineName, "-", Data.StageName)
| project TimeGenerated, deployment
| summarize count() by bin(TimeGenerated, 1d), deployment
| render columnchart


classis releases that were not successful
AzureDevOpsAuditing
| where OperationName == "Release.DeploymentCompleted" and Data.DeploymentResult != "Succeeded"
| extend deployment = strcat(Data.PipelineName, "-", Data.StageName)
| project TimeGenerated, deployment
| summarize count() by bin(TimeGenerated, 1d), deployment
| render columnchart

Azure Portal

RBAC

  • At any resource you can see which RBAC roles apply to this specific resource and what the allowed actions are: After selecting the resource, go to IAM - Roles → You'll now notice the roles that apply. Now, for any role, click on Details → This will show you the allowed actions and switch between the control and data plane for any data actions (useful with for example storage accounts).

GIT

Check git repository size
git count-objects -vH 


Combine fetch and merge to completely update a repository
git pull 

Log & Diff

Show git log
git log


Show the files which have changed between the branch and master
git diff --name-only master..branchtocheck

Branches

Create and immediately checkout a branch
git checkout -b "#123456-add-dns-resolving"


Show all branches
git branch -a


Show all active branches
git ls-remote --heads origin


Merge changes from master branch, perform from inside the branch
git merge master

Git Remotes

Show remote
git remote -v


Update the remote “origin” with a new url in case of a rename of the repo or project
git remote set-url origin https://getshifting@dev.azure.com/getshifting/Infra/_git/infraplayground

Commits

Add and commit a new file
git add .
git commit -m "Added new file"


Add and commit in 1 go for changed files
git commit -am "changed a file"


Co-Authoring / Pair coding commit message
Changed a couple of files

Co-Authored-By: Sjoerd <sjoerd @ getshifting.com>


Do not trigger CI in Azure DevOps
***NO_CI***

Databases

MS SQL

Start MS SQL Management Studio as a user from a different domain to a different database server
runas /netonly /user:getshifting.local\admin-sjoerd "c:\Program Files (x86)\Microsoft SQL Server Management Studio 18\Common7\IDE\Ssms.exe -S dbdev01.getshifting.local"

PostgreSQL

# get postgresql user and password
cd /data/docker
cat .env
docker container ps
docker exec -it container_db /bin/bash
psql -U <user> -W


See connections
SHOW max_connections;
SHOW shared_buffers;


See current activity
SELECT pid AS process_id, 
       usename AS username, 
       datname AS database_name, 
       client_hostname, 
       application_name,
       backend_start,
       state,
       wait_event,
       state_change
FROM pg_stat_activity;


See all databases
\l
-- See all databases with more information
\l+

Python

Python Pipeline Tasks

- task: PythonScript@0
  displayName: "Sort and print all system environments "
  inputs:
    scriptSource: 'inline'
    script: |
      import os
      for name, value in sorted(os.environ.items()):
        print("   " + name + "=" + value)

- task: PythonScript@0
  displayName: "Python version "
  inputs:
    scriptSource: 'inline'
    script: |
      import subprocess
      import os
      if os.environ['AGENT_OS'] == "Linux":
        version = subprocess.getoutput('python3 --version')
      elif os.environ['AGENT_OS'] == "Windows_NT":
        version = subprocess.getoutput('py -3 --version')
      else:
        version = "unknown"
      print(f"Python version is {version}")

- task: PythonScript@0
  displayName: "Pip version "
  inputs:
    scriptSource: 'inline'
    script: |
      import subprocess
      pip = subprocess.getoutput('pip --version')
      print(f"Pip version is {pip}")

- task: PowerShell@2
  displayName: "Python virtual environment"
  inputs:
    pwsh: true
    targetType: 'inline'
    script: |
      Write-Host "`n##[section]Start python script`n"
      # Create virtual environment "env" by running module venv (https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/)
      py -m venv env
      .\env\Scripts\activate.ps1 -Verbose
      Write-Host "`n##[section]Verify virtual environment`n"
      Get-Command python.exe
      Write-Host "`n##[section]Install packages`n"
      $scriptDir = "$(BUILD.SourcesDirectory)\$(BUILD.REPOSITORY.NAME)\src\$module"
      Write-Host "Locatie: $(Get-Location); Script dir: $scriptDir"
      if (Test-Path $scriptDir\requirements.txt){
          Write-Host "Directory: $scriptDir; Requirements file: Present"
          py -m pip install -r $scriptDir\requirements.txt
      }else {
          Write-Host "Directory: $scriptDir; Requirements file: Not present. Installing from code."
          # install a package 
          py -m pip install requests
          # install a specific version 
          py -m pip install pandas==1.5.2
      }
      Write-Host "`n##[section]Run python script`n"
      if (Test-Path $scriptDir\main.py){
          Write-Host "Script: $scriptDir\main.py; Status: Present"
          py $scriptDir\main.py
      }else{
          Throw "Script: not found in $scriptDir"
      }
 
      Write-Host "`n##[section]Install packages`n"
      # install a package 
      py -m pip install requests
      # install a specific version 
      py -m pip install pandas==1.5.2

AWS Lambda Node.js

Get Dynamo DB Table as Log

const AWS = require("aws-sdk");
const dynamodb = new AWS.DynamoDB();
 
// Get Time in the Netherlands
var dutchTime = new Date().toLocaleString('en-US', {timeZone: "Europe/Amsterdam"});
var requestTime = new Date(dutchTime);
var year = requestTime.getFullYear();
var month = requestTime.getMonth() + 1; //Months go from 0 - 11
// Set months to get logs for
if (month === 1){
    var fullMonth = year + "-0" + month;
    var lastMonth = (year - 1 ) + "-12";
}
if (month > 1 && month < 10){
    var fullMonth = year + "-0" + month;
    var lastMonth = year + "-0" + (month - 1);
}
if (month === 10){
    var fullMonth = year + "-" + month;
    var lastMonth = year + "-0" + (month - 1);
}
if (month > 10){
    var fullMonth = year + "-" + month;
    var lastMonth = year + "-" + (month - 1);
}
 
const params = {
  TableName: 'LogTable',
  FilterExpression: 'begins_with(RequestTime, :RequestTime) OR begins_with(RequestTime, :RequestTime2)',
  ExpressionAttributeValues: {
    ':RequestTime': {S: fullMonth},
    ':RequestTime2': {S: lastMonth},
  },
};
 
exports.handler = (event, context, callback) => {
    console.log(fullMonth + lastMonth);
    dynamodb.scan(params, (err, data) => {
        if(err){
            console.log('Error on retrieving log data ' + err);
        }
        callback(null, data['Items']);
    });
};

Send Email

//jshint esversion:6
 
/* IAM-Access
Description: Sends the end email to the customer with all required information. 
 
Links:
- AWS Javascript SDK for SES: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SES.html
- Util.inspect and JSON.stringify: https://medium.com/@reginald.johnson/testing-aws-lambda-functions-the-easy-way-41cf1ed8c090 
 
*/
 
// AWS Variables
var aws = require('aws-sdk');
aws.config.update({region: 'eu-west-1'});
// SES const & vars
const ses = new aws.SES();
var util = require('util');
var emailfrom = 'info@shiftwiki . nl'; 
 
// Workflow Variables
// Declaring the customerData variable
var customerData;
 
// Send email to customer
function sendMail(params){
  console.log("Start function sendMail. Email address: " + params.email );
  if (customerData.discountCode === "testmodus"){
    emailto = 'sjoerd@shiftwiki .nl';
    emailfrom = 'info@shiftwiki .nl';
  } else {
    emailto = params.email;
  }
  console.log("Email address: " + emailto );
  //email
  var mailparams = {
      Destination: {
          ToAddresses: [emailto]
      },
      Message: {
          Body: {
              Html: { 
                  Charset: "UTF-8",
                  Data: "Dear " + params.firstname + " " + params.lastname +
                  "<br><br>" +
                  "All is ready. " +
                  "<br><br>" +
                  "Kind regards, <br>" +
                  "Shift"
              }
          },
          Subject: { Data: "[SHIFT]Ready. "}
      },
      Source: emailfrom
  };
  return ses.sendEmail(mailparams).promise();
}
 
// Error Handling
function errorResponse(message) {
  this.name = "email-to-customer-end";
  this.message = message;
}
errorResponse.prototype = new Error();
 
exports.handler = (event, context, callback) => {
    console.log("Start email-to-customer-end");
    //console.log(event);
    //console.log(context);
    customerData = event;
 
    // Display available information
    console.log(customerData);
 
    // Invoke function sendmail
    sendMail(customerData).then(() => {
      customerData.status = customerData.status + "Customer end email sent out. ";
      callback(null, customerData); 
    }).catch((err) => {
      // Error to CloudWatch
      console.error(err); 
      // Add Error to Task Output
      const error = new errorResponse(err.message);
      callback(error);
    });       
};

Create S3 Bucket

//jshint esversion:6
 
/* S3-Buckets-Add
Description: Creates two S3 buckets, configure access policy, website hosting and redirect.
 
Original PowerShell Script:
### S3 Bucket Creation as a Static Website ###
New-S3Bucket -BucketName $bucketname -PublicReadOnly 
Write-S3BucketWebsite -BucketName $bucketname -WebsiteConfiguration_IndexDocumentSuffix index.html -WebsiteConfiguration_ErrorDocument error.html
$s3policy = '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::bucketnameplaceholder/*"}]}'
$s3policy = $s3policy -replace "bucketnameplaceholder", "$bucketname"
Write-S3BucketPolicy -BucketName "$bucketname" -Policy $s3policy
write-host "S3 Bucket $bucketname was created on $((Get-S3Bucket -BucketName $bucketname).creationdate) "
### S3 Bucket Creation Redirect ###
New-S3Bucket -BucketName $wwwbucketname
Add-S3PublicAccessBlock -BucketName $wwwbucketname `
-PublicAccessBlockConfiguration_BlockPublicAcl $true `
-PublicAccessBlockConfiguration_BlockPublicPolicy $true `
-PublicAccessBlockConfiguration_IgnorePublicAcl $true `
-PublicAccessBlockConfiguration_RestrictPublicBucket $true
Write-S3BucketWebsite -BucketName $wwwbucketname -RedirectAllRequestsTo_HostName $bucketname
write-host "S3 Bucket $wwwbucketname was created on $((Get-S3Bucket -BucketName $wwwbucketname).creationdate) "
 
Links:
- AWS Javascript SDK for S3: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
 
*/
 
// AWS Variables
var aws = require('aws-sdk');
aws.config.update({region: 'eu-west-1'});
var s3 = new aws.S3();
 
// Workflow Variables
// Declaring the customerData variable
var customerData;
 
//function s3BucketCreate
function s3BucketCreate(params) {
  console.log("Start function s3BucketCreate. Input: " + params.domainname);
  return s3.createBucket({
    Bucket: params.domainname, 
    CreateBucketConfiguration: {
     LocationConstraint: aws.config.region
    },
    ACL: "public-read"
  }).promise();
}
 
//function s3wwwBucketCreate
function s3wwwBucketCreate(params) {
  console.log("Start function s3wwwBucketCreate. Input: " + params.wwwdomainname);
  return s3.createBucket({
    Bucket: params.wwwdomainname, 
    CreateBucketConfiguration: {
     LocationConstraint: aws.config.region
    }
  }).promise();
}
 
//function s3BucketWebsite
function s3BucketWebsite(params){
  console.log("Start function s3BucketWebsite. Input: " + params.domainname);
  return s3.putBucketWebsite({
    Bucket: params.domainname,
    WebsiteConfiguration: { 
      ErrorDocument: {
        Key: 'error.html' 
      },
      IndexDocument: {
        Suffix: 'index.html' 
      }
    }
  }).promise();
}
 
//function s3BucketPolicy
function s3BucketPolicy(params){
  console.log("Start function s3BucketPolicy. Input: " + params.domainname);
  var s3policyplaceholder = "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"s3:GetObject\",\"Resource\":\"arn:aws:s3:::domainnameplaceholder/*\"}]}";
  var s3policy = s3policyplaceholder.replace("domainnameplaceholder", params.domainname);
  return s3.putBucketPolicy({
    Bucket: params.domainname, 
    Policy: s3policy,
  }).promise();
}
 
//function s3BucketPublicAccessBlock
function s3BucketPublicAccessBlock(params){
  console.log("Start function s3BucketPublicAccessBlock. Input: " + params.wwwdomainname);
  return s3.putPublicAccessBlock({
    Bucket: params.wwwdomainname,
    PublicAccessBlockConfiguration: { 
      BlockPublicAcls: true,
      BlockPublicPolicy: true,
      IgnorePublicAcls: true,
      RestrictPublicBuckets: true
    }
  }).promise();
}
 
//function s3BucketWebsiteRedirect
function s3BucketWebsiteRedirect(params){
  console.log("Start function s3BucketWebsiteRedirect. Input: " + params.wwwdomainname + " and " + params.domainname);
  return s3.putBucketWebsite({
    Bucket: params.wwwdomainname,
    WebsiteConfiguration: { 
      RedirectAllRequestsTo: {
        HostName: params.domainname
      }
    }
  }).promise();
}
 
// Error Handling
function errorResponse(message) {
  this.name = "s3-buckets-add";
  this.message = message;
}
errorResponse.prototype = new Error();
 
exports.handler = (event, context, callback) => {
    console.log("Start s3-buckets-add");
    // console.log(event);
    customerData = event;
 
    // Display available information
    console.log(customerData);
 
    // create params variable to pass to s3 functions
    var params = {
      domainname: customerData.domainname,
      wwwdomainname: customerData.wwwdomainname
    };
 
    // Invoke function s3BucketCreate
    s3BucketCreate(params).then(() => {
      customerData.status = customerData.status + "S3 Bucket creation success. ";
      s3BucketWebsite(params).then(() => {
          customerData.status = customerData.status + "S3 Bucket configured as website. ";
          s3BucketPolicy(params).then(() => {
            customerData.status = customerData.status + "S3 Bucket Policy set. ";
            s3wwwBucketCreate(params).then(() => {
              customerData.status = customerData.status + "S3 wwwBucket creation success. ";
              s3BucketPublicAccessBlock(params).then(() => {
                customerData.status = customerData.status + "S3 wwwBucket public access blocked. ";
                s3BucketWebsiteRedirect(params).then(() => {
                  customerData.status = customerData.status + "S3 wwwBucket redirected to S3 Bucket. ";
                  callback(null, customerData); 
                }).catch((err) => {
                  // Error to CloudWatch
                  console.error(err); 
                  // Add Error to Task Output
                  const error = new errorResponse(err.message);
                  callback(error);
                });
              }).catch((err) => {
                // Error to CloudWatch
                console.error(err); 
                // Add Error to Task Output
                const error = new errorResponse(err.message);
                callback(error);
              });
            }).catch((err) => {
              // Error to CloudWatch
              console.error(err); 
              // Add Error to Task Output
              const error = new errorResponse(err.message);
              callback(error);
            });
          }).catch((err) => {
            // Error to CloudWatch
            console.error(err); 
            // Add Error to Task Output
            const error = new errorResponse(err.message);
            callback(error); 
          });
      }).catch((err) => {
        // Error to CloudWatch
        console.error(err); 
        // Add Error to Task Output
        const error = new errorResponse(err.message);
        callback(error); 
      });
    }).catch((err) => {
      // Error to CloudWatch
      console.error(err); 
      // Add Error to Task Output
      const error = new errorResponse(err.message);
      callback(error);
    });
};

VSCode

Key Combinations

  • Comment / uncomment: ctrl + /
  • Word wrap: alt + z
  • toggle from editor to terminal: ctrl + `
  • toggle from terminal to editor: ctrl + 1

Replace (CTRL + H)

  • Tabs: ALT 009
You could leave a comment if you were logged in.
cheatsheet.txt · Last modified: 2024/04/02 22:25 by sjoerd