Issues with Verify script

Hello,
Thanks for sharing your scripts.
I have been trying to deploy the: AutoRestoreAndVerify script.
I don’t want to perform the Restore, just the Verify; so I have modified the script and perhaps this is the cause of my problems.
I’m hoping you can help me to troubleshoot and resolve the errors that I am observing.
The environment is:
Debian11 (Bullseye)
Bacula Community Edition 13.0.3
#!/bin/bash
#
# Verify.sh
#
# ------------------------------------------------------------------------------
# 20170831 - Changelog moved to bottom of script.
# ------------------------------------------------------------------------------
#
# BSD 2-Clause License
#
# Copyright (c) 2017-2023, William A. Arlofski waa-at-revpol-dot-com
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1.  Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# 2.  Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
 IN NO EVENT SHALL THE COPYRIGHT
# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# ------------------------------------------------------------------------------

# Set some variables
# ------------------
bcbin="/opt/bacula/bin/bconsole"
bccfg="/opt/bacula/etc/bconsole.conf"
cleanup="yes"  # Currently not used at the end... Too dangerous

# --------------------------------------------------
# Nothing should need to be modified below this line
# --------------------------------------------------

# ============================================================================================================
# -----------------------------------
# Example Catalog Backup Job snippet:
# -----------------------------------
#
# Job {
#   Name = Catalog
#   Client = bacula-fd
#   JobDefs = Defaults-To-Speedy
#   FileSet = Catalog
#   Level = Full
#   Schedule = Catalog
#   Priority = 20
#   WriteBootstrap = "/opt/bacula/bsr/%n_%i.bsr"
#
#   RunScript {
#     RunsWhen = before
#     RunsOnClient = no
#     FailJobOnError = yes
#     # waa - 20150215 - One custom RunsBefore script to dump a time-stamped
#     #                  bacula database file and then remove old copies
#     # --------------------------------------------------------------------
#     Command = "/opt/bacula/scripts/bacula_pgsqlbackup.sh"
#   }
#
# # waa - 20170502 - Run an Auto-restore and then run three Verify
# #                  jobs: VolumeToCatalog, DiskToCatalog, and Data
# # ---------------------------------------------------------------
#   RunScript {
#     RunsWhen = after
#     RunsOnClient = no
#     RunsOnFailure = no
#
#     # Call _this_ script and pass the current backup job jobid, client, jobname, and fileset
#     # --------------------------------------------------------------------------------------
#     Command = "/opt/bacula/scripts/AutoRestoreAndVerify.sh %i %c %n %f"
#   }
# }
#
# -------------------------------------------------------------------------
# Example "special" RestoreJob with same priority as our Catalog Backup Job
# -------------------------------------------------------------------------
# Job {
#   Name = RestoreCatalog
#   Type = Restore
#   Messages = Standard
#   Where = /tmp/bacula-auto_restores
#   MaximumConcurrentJobs = 10
#   WriteBootstrap = "/opt/bacula/bsr/%c_%n_%i.bsr"
#   Priority = 20     # waa - 20170802 - Set to match the same priority as the Catalog job to prevent dead lock
#   # I have created a special Storage, Client, FileSet and Pool, each called "None"
#   # for use in Copy/migration jobs, or in this case for my special Catalog Restore Job
#   # ----------------------------------------------------------------------------------
#   Storage  = None   # waa - 20150715 - Is never used, but required by Bacula config parser
#   Client   = None   # waa - 20131020 - Is never used, but required by Bacula config parser
#   FileSet  = None   # waa - 20131020 - Is never used, but required by Bacula config parser
#   Pool     = None   # waa - 20131020 - Is never used, but required by Bacula config parser
# }
# ============================================================================================================

# -------------------------------------------------------------------------
# When called from a RunScript{} (RunsWhen = After) in a Backup Job, we get
# %i (jobid), %c (client), %n (jobname), and %f (fileset) as $1, $2, $3, $4
# -------------------------------------------------------------------------

# Simple test to verify at least four command line arguments were supplied
# ------------------------------------------------------------------------
if [ $# -lt 4 ]; then
  echo -e "\nUse: $0 <jobid> <client> <jobname> <fileset>"
  echo -e "Command line received: $0 $@\n"
  exit 1
fi

# Verify that the bconsole config file exists
# -------------------------------------------
if [ ! -e ${bccfg} ]; then
  echo -e "\nThe bconsole configuration file does not seem to be '${bccfg}'."
  echo -e "Please check the setting for the variable 'bccfg'.\n"
  exit 1
fi

# Verify that the bconsole binary exists and is executable
# --------------------------------------------------------
if [ ! -x ${bcbin} ]; then
  echo -e "\nThe bconsole binary does not seem to be '${bcbin}', or it is not executable."
  echo -e "Please check the setting for the variable 'bcbin'.\n"
  exit 1
fi

# Create the temporary restore directory including the Job name
# -------------------------------------------------------------
# restoredir=$(mktemp -q -d /mnt/iscsi_backups/bacula/bacula-auto_restores-$3-XXXX)

# Start the Restore job and get its jobid so we can wait for it to finish
# Restore jobs currently do not accept the "priority" option on the command line!
# So, we need to create a special "RestoreCatalog" restore job resource and set its priority=20 (Same as our Catalog job)
# so it can be triggered in a RunScript (RunsWhen = After) and not be held up waiting for: "higher priority jobs to finish"
# The "higher priority Job" being the actual Catalog backup job. We would end up in a deadlock in this case.
# -------------------------------------------------------------------------------------------------------------------------
#echo "Starting restore of backup job \"$3\" (jobid $1) to restore directory \"${restoredir}\""
#status=$(echo "restore jobid=$1 client=$2 restoreclient=$2 restorejob=RestoreCatalog \
#              where=${restoredir} comment=\"AutoRestore of job: $3 to ${restoredir}\" all done yes" \
#               | ${bcbin} -c ${bccfg})

# Get the jobid of the Restore job and print it
# ---------------------------------------------
#jobid=$(echo "${status}" | grep "^Job queued" | cut -d'=' -f2)
#echo "Restore Job's jobid=${jobid}"

# Wait for the Restore job to finish, then start the VolumeToCatalog Verify job
# -----------------------------------------------------------------------------
#echo "Queueing VolumeToCatalog verify of job \"$3\", (jobid $1)"
status=$(echo -e "run job=Verify_Catalog jobid=$1 client=$2 level=VolumeToCatalog \
                 comment=\"VolumeToCatalog of job: $3\" yes" | ${bcbin} -c ${bccfg})

# Get the jobid of the VolumeToCatalog job and print it
# -----------------------------------------------------
jobid=$(echo "${status}" | grep "^Job queued" | cut -d'=' -f2)
echo "VolumeToCatalog Job's jobid=${jobid}"

# Wait for the VolumeToCatalog Verify job to finish, then start the DiskToCatalog Verify job
# NOTE: the "fileset =" is required in the named job resource, or on the command line!
# ------------------------------------------------------------------------------------------
echo "Queueing DiskToCatalog verify of job \"$3\", (jobid $1)"
status=$(echo -e "wait jobid=${jobid} \n run job=Verify_Catalog jobid=$1 client=$2 fileset=$4 level=DiskToCatalog \
                  comment=\"DiskToCatalog of job: $3\" yes" | ${bcbin} -c ${bccfg})

# Get the jobid of the DiskToCatalog job and print it
# ---------------------------------------------------
jobid=$(echo "${status}" | grep "^Job queued" | cut -d'=' -f2)
echo "DiskToCatalog Job's jobid=${jobid}"

# Wait for the DiskToCatalog Verify job to finish, then start the Level=Data Verify job
# NOTE: the "fileset =" is required in the named job resource, or on the command line!
# -------------------------------------------------------------------------------------
echo "Queueing DATA verify of job \"$3\", (jobid $1)"
status=$(echo -e "wait jobid=${jobid} \n run job=Verify_Catalog jobid=$1 client=$2 fileset=$4 level=Data \
                  comment=\"Data Verify of job: $3\" yes" | ${bcbin} -c ${bccfg})

# Get the jobid of the DATA Verification job and print it
# -------------------------------------------------------
jobid=$(echo "${status}" | grep "^Job queued" | cut -d'=' -f2)
echo "Data Verify Job's jobid=${jobid}"

# Wait for the DATA Verify job to finish
# --------------------------------------
echo "Waiting for DATA verify job (jobid ${jobid}) to finish..."
status=$(echo -e "wait jobid=${jobid} \n" | ${bcbin} -c ${bccfg})

# Delete the temporary restore directory?
# ---------------------------------------
# if [ "${cleanup}" = "yes" ]; then
#   echo "Deleting temporary restore directory '${restoredir}'..."
#   rm -rf ${restoredir}

echo -e "\nFinished...\n"
# -------------
# End of script
# -------------

# ----------
# Change Log
# ----------
# ----------------------------
# William A. Arlofski
#<EMAIL_ADDRESS># ----------------------------
#
# 20170831 - Initial release
#            Automatically restore a backup job, then run all three
#            verification jobs against it
#          - May be run manully or triggered in a RunScript of a job
#
# 20170902 - Added some more logging
#
# 20190129 - Changed the 'restoredir' variable to be randomly created using the
#            mktemp utility
#          - No longer attempt to "rm -rf" a statically named restore directory
# 20230304 - Added Job name to the 'restoredir' variable
# 20230614 - Minor clean up before releasing on Github.
# --------------------------------------------------------------------------------

Job {
  Name = "debian11-DR6-backup"
  JobDefs = "DefaultJob"
  Runscript {
    RunsOnClient = no
    RunsWhen = "After"
    Command = "/opt/bacula/scripts/verify.sh %i %c %n %f"
  }
  Accurate = yes
}

JobDefs {
  Name = "DefaultJob"
  Type = "Backup"
  Level = "Incremental"
  Messages = "Standard"
  Storage = "File1"
  Pool = "File"
  Client = "debian11-DR6-fd"
  Fileset = "Full Set"
  Schedule = "WeeklyCycle"
  WriteBootstrap = "/opt/bacula/working/%c.bsr"
  SpoolAttributes = yes
  Pr`

Here is the joblog output
Log from: verify-Data
debian11-DR6-dir JobId 195: Error: Bacula Enterprise debian11-DR6-dir 13.0.3 (02May23):
  Build OS:               x86_64-pc-linux-gnu-bacula-enterprise debian 11.2
  JobId:                  195
  Job:                    debian11-DR6-verify-Data.2023-11-01_23.05.00_37
  FileSet:                Catalog
  Verify Level:           Incremental
  Client:                 debian11-DR6-fd
  Verify JobId:           0
  Verify Job:             debian11-DR6-backup
  Start time:             01-Nov-2023 23:05:00
  End time:               01-Nov-2023 23:05:00
  Elapsed time:           1 sec
  Files Examined:         0
  Non-fatal FD errors:    1
  FD termination status:
  Termination:            *** Verify Error ***
debian11-DR6-dir JobId 195: Fatal error: verify.c:72 Unimplemented Verify level 73(I)

Log from: verify-DiskToCatalog
debian11-DR6-dir JobId 196: Error: Bacula Enterprise debian11-DR6-dir 13.0.3 (02May23):
  Build OS:               x86_64-pc-linux-gnu-bacula-enterprise debian 11.2
  JobId:                  196
  Job:                    debian11-DR6-verify-DiskToCatalog.2023-11-01_23.05.00_39
  FileSet:                Catalog
  Verify Level:           Incremental
  Client:                 debian11-DR6-fd
  Verify JobId:           0
  Verify Job:             debian11-DR6-backup
  Start time:             01-Nov-2023 23:05:00
  End time:               01-Nov-2023 23:05:00
  Elapsed time:           1 sec
  Files Examined:         0
  Non-fatal FD errors:    1
  FD termination status:
  Termination:            *** Verify Error ***
debian11-DR6-dir JobId 196: Fatal error: verify.c:72 Unimplemented Verify level 73(I)

Log from: verify-VolumeToCatalog
debian11-DR6-dir JobId 197: Error: Bacula Enterprise debian11-DR6-dir 13.0.3 (02May23):
  Build OS:               x86_64-pc-linux-gnu-bacula-enterprise debian 11.2
  JobId:                  197
  Job:                    debian11-DR6-verify-VolumeToCatalog.2023-11-01_23.05.00_41
  FileSet:                Catalog
  Verify Level:           Incremental
  Client:                 debian11-DR6-fd
  Verify JobId:           0
  Verify Job:             debian11-DR6-backup
  Start time:             01-Nov-2023 23:05:00
  End time:               01-Nov-2023 23:05:00
  Elapsed time:           1 sec
  Files Examined:         0
  Non-fatal FD errors:    1
  FD termination status:
  Termination:            *** Verify Error ***
debian11-DR6-dir JobId 197: Fatal error: verify.c:72 Unimplemented Verify level 73(I)

Hello,
Can you show a joblog of the debian11-DR6-backup which calls the Verify.sh script?
In bconsole, please do:

ll joblog jobid=xxxx (where xxxx is a jobid of one of this jobs recent runs)

list job=debian11-DR6-backup   will show all the runs of this job

`*ll joblog jobid=194
time: 2023-11-01 23:05:02
logtext: debian11-DR6-dir JobId 194: Start Backup JobId 194, Job=debian11-DR6-backup.2023-11-01_23.05.00_36
time: 2023-11-01 23:05:02

logtext: debian11-DR6-dir JobId 194: Connected to Storage "File1" at debian11-DR6:9103 with TLS
time: 2023-11-01 23:05:02

logtext: debian11-DR6-dir JobId 194: Using Device "FileChgr1-Dev1" to write.
time: 2023-11-01 23:05:02

logtext: debian11-DR6-dir JobId 194: Connected to Client "debian11-DR6-fd" at debian11-DR6:9102 with TLS
time: 2023-11-01 23:05:02

logtext: debian11-DR6-fd JobId 194: Connected to Storage at debian11-DR6:9103 with TLS
time: 2023-11-01 23:05:02

logtext: debian11-DR6-dir JobId 194: Sending Accurate information to the FD.
time: 2023-11-01 23:05:04

logtext: debian11-DR6-sd JobId 194: Volume "Vol-0001" previously written, moving to end of data.
time: 2023-11-01 23:05:04

logtext: debian11-DR6-sd JobId 194: Ready to append to end of Volume "Vol-0001" size=62,489,393,221
time: 2023-11-01 23:05:17

logtext: debian11-DR6-sd JobId 194: Elapsed time=00:00:13, Transfer rate=72.55 M Bytes/second
time: 2023-11-01 23:05:17

logtext: debian11-DR6-sd JobId 194: Sending spooled attrs to the Director. Despooling 1,965,929 bytes ...
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: Bacula Enterprise debian11-DR6-dir 13.0.3 (02May23):
Build OS:               x86_64-pc-linux-gnu-bacula-enterprise debian 11.2
JobId:                  194
Job:                    debian11-DR6-backup.2023-11-01_23.05.00_36
Backup Level:           Incremental, since=2023-10-31 23:05:02
Client:                 "debian11-DR6-fd" 13.0.3 (02May23) x86_64-pc-linux-gnu-bacula-enterprise,debian,11.2
FileSet:                "Full Set" 2023-10-26 10:24:36
Pool:                   "File" (From Job resource)
Catalog:                "MyCatalog" (From Client resource)
Storage:                "File1" (From Job resource)
Scheduled time:         01-Nov-2023 23:05:00
Start time:             01-Nov-2023 23:05:02
End time:               01-Nov-2023 23:05:17
Elapsed time:           15 secs
Priority:               10
FD Files Written:       9,694
SD Files Written:       9,694
FD Bytes Written:       941,790,263 (941.7 MB)
SD Bytes Written:       943,213,328 (943.2 MB)
Rate:                   62786.0 KB/s
Software Compression:   None
Comm Line Compression:  63.9% 2.8:1
Snapshot/VSS:           no
Encryption:             no
Accurate:               yes
Volume name(s):         Vol-0001
Volume Session Id:      26
Volume Session Time:<PHONE_NUMBER>
Last Volume Bytes:      63,433,474,849 (63.43 GB)
Non-fatal FD errors:    0
SD Errors:              0
FD termination status:  OK
SD termination status:  OK
Termination:            Backup OK
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: Begin pruning Jobs older than 3 years .
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: No Jobs found to prune.
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: Begin pruning Files.
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: No Files found to prune.
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: End auto prune.
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: shell command: run AfterJob "/opt/bacula/scripts/verify.sh 194 debian11-DR6-fd debian11-DR6-backup Full Set"
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: AfterJob: VolumeToCatalog Job's jobid=
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: AfterJob: Queueing DiskToCatalog verify of job "debian11-DR6-backup", (jobid 194)
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: AfterJob: DiskToCatalog Job's jobid=
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: AfterJob: Queueing DATA verify of job "debian11-DR6-backup", (jobid 194)
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: AfterJob: Data Verify Job's jobid=
time: 2023-11-01 23:05:17

logtext: debian11-DR6-dir JobId 194: AfterJob: Waiting for DATA verify job (jobid ) to finish...
time: 2023-11-01 23:05:18

logtext: debian11-DR6-dir JobId 194: AfterJob:
time: 2023-11-01 23:05:18

logtext: debian11-DR6-dir JobId 194: AfterJob: Finished...
time: 2023-11-01 23:05:18

logtext: debian11-DR6-dir JobId 194: AfterJob:
       jobid: 194
         job: debian11-DR6-backup.2023-11-01_23.05.00_36
        name: debian11-DR6-backup
 purgedfiles: 0
        type: B
       level: I
    clientid: 1
  clientname: debian11-DR6-fd
   jobstatus: T
   schedtime: 2023-11-01 23:05:00
   starttime: 2023-11-01 23:05:02
     endtime: 2023-11-01 23:05:17
 realendtime: 2023-11-01 23:05:17
    jobtdate: 1,698,840,317
volsessionid: 26

volsessiontime: 1,698,296,880
jobfiles: 9,694
jobbytes: 941,790,263
readbytes: 941,790,263
joberrors: 0
jobmissingfiles: 0
poolid: 2
poolname: File
priorjobid: 0
priorjob:
filesetid: 7
fileset: Full Set
hascache: 0
comment:
reviewed: 0

`

Thank you.
I think I see a possible problem.
The older default example configurations ship with a terribly named FileSet called Full Set (yes, with a space in the name!)  :(
When sent to the script as %f, becoming $4 inside the script the $4 becomes just 'Full' and 'Set' would be $5, but the script is not expecting spaces in variables.
I would ask you to do the following:

Rename the Fileset "Full Set" to ANYTHING having no spaces in the name, like "FullSet" or "Full_Set"
Edit all of your jobs using this Fileset
Test your config
Reload the director config
Re run this job that calls the Verify.sh script
Report the results

If you are still having issues, please look at the joblog of the backup job that calls this script again, then grab the "run AfterJob" line and run it manually like:
bash -x /opt/bacula/scripts/verify.sh xxxx debian11-DR6-fd debian11-DR6-backup yyyy
'''
Where xxxx is the jobid of a new run of this job after modifying the FileSet, and yyyy is the new FileSet's name

P.S. When posting code, or logs, please use three backticks instead of one. It formats it as code in monospaced text. I have been editing your posts to make them more readable. :)

Good luck!

And there is one more thing...
My script is meant to call ONE Verify job several times using different verify levels.
Some things I see in your initial post are showing that you are calling a few different verify job names debian11-DR6-verify-Data.,  debian11-DR6-verify-DiskToCatalog, and debian11-DR6-verify-VolumeToCatalog which seem to have a "level" of  Incremental which is not a valid level for a Verify job.
I will probably need to see that bash -x  full script output and possibly your full configuration.

Hello Bill,
Thanks for your help.
I had to run the script manually. Hopefully I've got the formatting right this time.
bash -x /opt/bacula/scripts/verify.sh 199 debian11-DR6-fd debian11-DR6-backup Full_Set
+ bcbin=/opt/bacula/bin/bconsole
+ bccfg=/opt/bacula/etc/bconsole.conf
+ cleanup=yes
+ '[' 4 -lt 4 ']'
+ '[' '!' -e /opt/bacula/etc/bconsole.conf ']'
+ '[' '!' -x /opt/bacula/bin/bconsole ']'
++ /opt/bacula/bin/bconsole -c /opt/bacula/etc/bconsole.conf
++ echo -e 'run job=Verify_Catalog jobid=199 client=debian11-DR6-fd level=VolumeToCatalog                  comment="VolumeToCatalog of job: debian11-DR6-backup" yes'
+ status='Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
run job=Verify_Catalog jobid=199 client=debian11-DR6-fd level=VolumeToCatalog                  comment="VolumeToCatalog of job: debian11-DR6-backup" yes
Using Catalog "MyCatalog"
Job "Verify_Catalog" not found
The defined Job resources are:
     1: BackupCatalog
     2: RestoreFiles
     3: debian11-DR6-backup
     4: debian11-DR6-verify-Data
     5: debian11-DR6-verify-DiskToCatalog
     6: debian11-DR6-verify-VolumeToCatalog
Select Job resource (1-6): '
++ cut -d= -f2
++ grep '^Job queued'
++ echo 'Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
run job=Verify_Catalog jobid=199 client=debian11-DR6-fd level=VolumeToCatalog                  comment="VolumeToCatalog of job: debian11-DR6-backup" yes
Using Catalog "MyCatalog"
Job "Verify_Catalog" not found
The defined Job resources are:
     1: BackupCatalog
     2: RestoreFiles
     3: debian11-DR6-backup
     4: debian11-DR6-verify-Data
     5: debian11-DR6-verify-DiskToCatalog
     6: debian11-DR6-verify-VolumeToCatalog
Select Job resource (1-6): '
+ jobid=
+ echo 'VolumeToCatalog Job'\''s jobid='
VolumeToCatalog Job's jobid=
+ echo 'Queueing DiskToCatalog verify of job "debian11-DR6-backup", (jobid 199)'
Queueing DiskToCatalog verify of job "debian11-DR6-backup", (jobid 199)
++ /opt/bacula/bin/bconsole -c /opt/bacula/etc/bconsole.conf
++ echo -e 'wait jobid= \n run job=Verify_Catalog jobid=199 client=debian11-DR6-fd fileset=Full_Set level=DiskToCatalog                   comment="DiskToCatalog of job: debian11-DR6-backup" yes'
+ status='Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
ERR: Job was not found
 run job=Verify_Catalog jobid=199 client=debian11-DR6-fd fileset=Full_Set level=DiskToCatalog                   comment="DiskToCatalog of job: debian11-DR6-backup" yes
Job "Verify_Catalog" not found
The defined Job resources are:
     1: BackupCatalog
     2: RestoreFiles
     3: debian11-DR6-backup
     4: debian11-DR6-verify-Data
     5: debian11-DR6-verify-DiskToCatalog
     6: debian11-DR6-verify-VolumeToCatalog
Select Job resource (1-6): '
++ cut -d= -f2
++ grep '^Job queued'
++ echo 'Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
ERR: Job was not found
 run job=Verify_Catalog jobid=199 client=debian11-DR6-fd fileset=Full_Set level=DiskToCatalog                   comment="DiskToCatalog of job: debian11-DR6-backup" yes
Job "Verify_Catalog" not found
The defined Job resources are:
     1: BackupCatalog
     2: RestoreFiles
     3: debian11-DR6-backup
     4: debian11-DR6-verify-Data
     5: debian11-DR6-verify-DiskToCatalog
     6: debian11-DR6-verify-VolumeToCatalog
Select Job resource (1-6): '
+ jobid=
+ echo 'DiskToCatalog Job'\''s jobid='
DiskToCatalog Job's jobid=
+ echo 'Queueing DATA verify of job "debian11-DR6-backup", (jobid 199)'
Queueing DATA verify of job "debian11-DR6-backup", (jobid 199)
++ /opt/bacula/bin/bconsole -c /opt/bacula/etc/bconsole.conf
++ echo -e 'wait jobid= \n run job=Verify_Catalog jobid=199 client=debian11-DR6-fd fileset=Full_Set level=Data                   comment="Data Verify of job: debian11-DR6-backup" yes'
+ status='Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
ERR: Job was not found
 run job=Verify_Catalog jobid=199 client=debian11-DR6-fd fileset=Full_Set level=Data                   comment="Data Verify of job: debian11-DR6-backup" yes
Job "Verify_Catalog" not found
The defined Job resources are:
     1: BackupCatalog
     2: RestoreFiles
     3: debian11-DR6-backup
     4: debian11-DR6-verify-Data
     5: debian11-DR6-verify-DiskToCatalog
     6: debian11-DR6-verify-VolumeToCatalog
Select Job resource (1-6): '
++ cut -d= -f2
++ grep '^Job queued'
++ echo 'Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
ERR: Job was not found
 run job=Verify_Catalog jobid=199 client=debian11-DR6-fd fileset=Full_Set level=Data                   comment="Data Verify of job: debian11-DR6-backup" yes
Job "Verify_Catalog" not found
The defined Job resources are:
     1: BackupCatalog
     2: RestoreFiles
     3: debian11-DR6-backup
     4: debian11-DR6-verify-Data
     5: debian11-DR6-verify-DiskToCatalog
     6: debian11-DR6-verify-VolumeToCatalog
Select Job resource (1-6): '
+ jobid=
+ echo 'Data Verify Job'\''s jobid='
Data Verify Job's jobid=
+ echo 'Waiting for DATA verify job (jobid ) to finish...'
Waiting for DATA verify job (jobid ) to finish...
++ /opt/bacula/bin/bconsole -c /opt/bacula/etc/bconsole.conf
++ echo -e 'wait jobid= \n'
+ status='Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
ERR: Job was not found'
+ echo -e '\nFinished...\n'

Finished...

OK, I see now...
And, it looks like I somehow have accidentally removed a part of the instructions at the top..
Look, my script calls a Job named Verify_Catalog which does not exist in your configurations.
run job=Verify_Catalog jobid=199 client=debian11-DR6-fd level=VolumeToCatalog                  comment="VolumeToCatalog of job: debian11-DR6-backup" yes
Using Catalog "MyCatalog"
Job "Verify_Catalog" not found

I am not sure how notes about this are currently missing from the instructions at the top.
Since I use this full AutoRestoreAndVerify.sh script on my catalog job, I named my Verify Job Verify_Catalog. You can name it anything you like, but then remember to edit the script on th elines it is called and modify it to match.
My Verify_Catalog Verify Job  resource:
Job {
  Name = Verify_Catalog
  Enabled = yes
  Type = Verify
  Priority = 20                            # Must be same as the Catalog job so this will not be held "waiting for higher priorty..."
  Level = VolumeToCatalog     # Place holder. Level will be specified on the command line
  JobDefs = Defaults
  Client = None
  FileSet = None
  Pool = None
  Storage = speedy-file       # Must be correct, or specified on the command line
  Schedule = Manual           # to call this verify job. See the Catalog job and this script for examples
  MaximumConcurrentJobs = 5
  AllowDuplicateJobs = yes
}

I think this is the missing piece of the puzzle. I need to add this Verify Job example back into the instructions at the top.
You will need to set the level to 10 probably to match your "normal" backup job priority.
P.S. My Client, Pool, and FileSet of 'None' in this example Verify Job all exist in my configs and are place holders I use for Copy and Migration jobs where no real client, fileset, or pool is used. Some of this is overridden when the script calls the Verify_catalog job.
Hope this helps!

Example Verify_Catalog Verify Job resource re-added to the comments/instructions at the top of the script.

Hello Bill,
I've modified the original script and am running it now.
The first verify job has completed successfully, so I'm definitely on the right track.
I must admit, I was a little confused about how the backup jobs that I had created earlier and your script integrated, which ultimately was the issue here, as I didn't fully understand how to configure the script for my environment.
In addition, I couldn't figure out how to run the script manually (even though I had tried doing this) so that I could see what was happening when the script was executing.
Once I could see the bacula response to the script, then it became clear what the issues were.
Thanks for your help and thanks for sharing your scripts!

That's great to hear.
Glad you got it working.
I'll go ahead and close this issue.
Best regards,
Bill

Hello Bill,
If I want to schedule a copy job to run after the verify jobs, should I be creating a copy job and then adding code to the verify script to call the copy job?
Or can I use these directives in the backup job?
@.***
Regards,
Vaughan
From: Bill Arlofski @.>
Sent: Thursday, November 2, 2023 3:40 PM
To: waa/AutoRestoreAndVerify @.>
Cc: Vaughan Wickham @.>; Author @.>
Subject: Re: [waa/AutoRestoreAndVerify] Issues with Verify script (Issue #1)
That's great to hear.
Glad you got it working.
I'll go ahead and close this issue.
Best regards,
Bill
—
Reply to this email directly, view it on GitHubhttps://github.com/waa/AutoRestoreAndVerify/issues/1#issuecomment-1790053982, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOZVBH4DY4XTVAG46SXRJJ3YCMP25AVCNFSM6AAAAAA62EQ6X6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJQGA2TGOJYGI.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Hello Bill,
Unfortunately there is still an issue.
When I run the verify.sh manually using bash -x and the arguments, the updated script works fine.
See Job History here

However, when the scheduled backup job runs, the script fails.
I can see the reason, the "Level". When the script fails, the Level for the verify jobs: Incremental

I can't figure out where I need to change the configuration to fix this.

You are still running those three different Verify jobs somehow...
If you want to use my script (your modified version of it), it should be run from the Backup job.
The Verify.sh script calls ONE Verify Job with different levels. You should not be running those jobs three long-named Verify Jobs, they should be disabled or removed from your config and you should have the one 'Veriy_Catalog'  (or whatever you want to call it as I mentioned previously.

Hello Bill,
I think I should start again.
And not use my verify job, since it is not working correctly (although ironically it does work when I run it manually).
Can I just confirm my understanding of the requirements, as it is different to what I expected.
So, do I only create one Verify job?
The: Verify Catalog job
And from that the script will run the 3 verify jobs?

verify-VolumeToCatalog
verify-DiskToCatalog
verify-Data

If my understanding is correct, the only change I will make to your script will be to disable the restore, as I don't want that.
But apart from that change, I will be using your script as-is; so that will make any troubleshooting easier.
In fact if you were to update the script with a switch to disable / enable the restore. Then I could set restore to NO and I would be using your script as released.

Hello Bill,
I have replaced my version of the verify script, with your version, just commenting out the restore commands.
I deleted the verify jobs and created a verify job using your template
Job {
  Name = "Verify_Catalog"
  Description = "Verify backup - debian11-DR6"
  Type = "Verify"
  Level = "VolumeToCatalog"
  JobDefs = "DefaultJob"
  MaximumConcurrentJobs = 5
}

When I run the backup job, the verify jobs don't appear to run.
In this screenshot, reading from the bottom to the top:
I ran an incremental backup
The verify jobs appeared to hang, so I cancelled them.
Then the verify jobs that run, are where I used bash -x and called the script myself (log below)
Then tried another backup job, and first verify job hanging.

When I run the verify script manually, the verify jobs do run.
bash -x /opt/bacula/scripts/verify.sh 224 debian11-DR6-fd debian11-DR6-backup Full_Set
+ bcbin=/opt/bacula/bin/bconsole
+ bccfg=/opt/bacula/etc/bconsole.conf
+ cleanup=yes
+ '[' 4 -lt 4 ']'
+ '[' '!' -e /opt/bacula/etc/bconsole.conf ']'
+ '[' '!' -x /opt/bacula/bin/bconsole ']'
++ /opt/bacula/bin/bconsole -c /opt/bacula/etc/bconsole.conf
++ echo -e 'run job=Verify_Catalog jobid=224 client=debian11-DR6-fd level=VolumeToCatalog                  comment="VolumeToCatalog of job: debian11-DR6-backup" yes'
+ status='Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
run job=Verify_Catalog jobid=224 client=debian11-DR6-fd level=VolumeToCatalog                  comment="VolumeToCatalog of job: debian11-DR6-backup" yes
Using Catalog "MyCatalog"
Job queued. JobId=229'
++ cut -d= -f2
++ grep '^Job queued'
++ echo 'Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
run job=Verify_Catalog jobid=224 client=debian11-DR6-fd level=VolumeToCatalog                  comment="VolumeToCatalog of job: debian11-DR6-backup" yes
Using Catalog "MyCatalog"
Job queued. JobId=229'
+ jobid=229
+ echo 'VolumeToCatalog Job'\''s jobid=229'
VolumeToCatalog Job's jobid=229
+ echo 'Queueing DiskToCatalog verify of job "debian11-DR6-backup", (jobid 224)'
Queueing DiskToCatalog verify of job "debian11-DR6-backup", (jobid 224)
++ /opt/bacula/bin/bconsole -c /opt/bacula/etc/bconsole.conf
++ echo -e 'wait jobid=229 \n run job=Verify_Catalog jobid=224 client=debian11-DR6-fd fileset=Full_Set level=DiskToCatalog                   comment="DiskToCatalog of job: debian11-DR6-backup" yes'
+ status='Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=229
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
JobId=229
JobStatus=OK (T)
You have messages.
 run job=Verify_Catalog jobid=224 client=debian11-DR6-fd fileset=Full_Set level=DiskToCatalog                   comment="DiskToCatalog of job: debian11-DR6-backup" yes
Job queued. JobId=230'
++ cut -d= -f2
++ grep '^Job queued'
++ echo 'Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=229
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
JobId=229
JobStatus=OK (T)
You have messages.
 run job=Verify_Catalog jobid=224 client=debian11-DR6-fd fileset=Full_Set level=DiskToCatalog                   comment="DiskToCatalog of job: debian11-DR6-backup" yes
Job queued. JobId=230'
+ jobid=230
+ echo 'DiskToCatalog Job'\''s jobid=230'
DiskToCatalog Job's jobid=230
+ echo 'Queueing DATA verify of job "debian11-DR6-backup", (jobid 224)'
Queueing DATA verify of job "debian11-DR6-backup", (jobid 224)
++ /opt/bacula/bin/bconsole -c /opt/bacula/etc/bconsole.conf
++ echo -e 'wait jobid=230 \n run job=Verify_Catalog jobid=224 client=debian11-DR6-fd fileset=Full_Set level=Data                   comment="Data Verify of job: debian11-DR6-backup" yes'
+ status='Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=230
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
JobId=230
JobStatus=Differences (D)
You have messages.
 run job=Verify_Catalog jobid=224 client=debian11-DR6-fd fileset=Full_Set level=Data                   comment="Data Verify of job: debian11-DR6-backup" yes
Job queued. JobId=231'
++ cut -d= -f2
++ grep '^Job queued'
++ echo 'Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=230
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
JobId=230
JobStatus=Differences (D)
You have messages.
 run job=Verify_Catalog jobid=224 client=debian11-DR6-fd fileset=Full_Set level=Data                   comment="Data Verify of job: debian11-DR6-backup" yes
Job queued. JobId=231'
+ jobid=231
+ echo 'Data Verify Job'\''s jobid=231'
Data Verify Job's jobid=231
+ echo 'Waiting for DATA verify job (jobid 231) to finish...'
Waiting for DATA verify job (jobid 231) to finish...
++ /opt/bacula/bin/bconsole -c /opt/bacula/etc/bconsole.conf
++ echo -e 'wait jobid=231 \n'
+ status='Connecting to Director debian11-DR6:9101
1000 OK: 10002 debian11-DR6-dir Version: 13.0.3 (02 May 2023)
Enter a period to cancel a command.
wait jobid=231
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
JobId=231
JobStatus=OK (T)
You have messages.'
+ echo -e '\nFinished...\n'

Finished...

Hello,
Info from bconsole.
*list job=Verify_Catalog
+-------+----------------+---------------------+------+-------+----------+-------------+-----------+
| jobid | name           | starttime           | type | level | jobfiles | jobbytes    | jobstatus |
+-------+----------------+---------------------+------+-------+----------+-------------+-----------+
|   222 | Verify_Catalog | 2023-11-04 13:11:19 | V    | O     |        0 |           0 | A         |
|   223 | Verify_Catalog | 2023-11-04 13:12:30 | V    | d     |        0 |           0 | A         |
|   225 | Verify_Catalog | 2023-11-04 13:12:45 | V    | A     |        0 |           0 | A         |
|   226 | Verify_Catalog | 2023-11-04 13:21:50 | V    | O     |        0 |           0 | A         |
|   227 | Verify_Catalog | 2023-11-04 13:21:53 | V    | d     |        0 |           0 | A         |
|   228 | Verify_Catalog | 2023-11-04 13:21:56 | V    | A     |        0 |           0 | A         |
|   229 | Verify_Catalog | 2023-11-04 13:24:42 | V    | O     |      271 |           0 | T         |
|   230 | Verify_Catalog | 2023-11-04 13:24:49 | V    | d     |  252,272 |           0 | D         |
|   231 | Verify_Catalog | 2023-11-04 13:32:18 | V    | A     |      271 | 878,577,228 | T         |
|   233 | Verify_Catalog | 2023-11-04 13:40:43 | V    | O     |        0 |           0 | C         |
+-------+----------------+---------------------+------+-------+----------+-------------+-----------+
*ll joblog jobid=233
No results to list.
           jobid: 233
             job: Verify_Catalog.2023-11-04_13.40.43_06
            name: Verify_Catalog
     purgedfiles: 0
            type: V
           level: O
        clientid: 1
      clientname: debian11-DR6-fd
       jobstatus: C
       schedtime: 2023-11-04 13:40:43
       starttime: 2023-11-04 13:40:43
         endtime:
     realendtime:
        jobtdate: 1,699,065,643
    volsessionid: 0
  volsessiontime: 0
        jobfiles: 0
        jobbytes: 0
       readbytes: 0
       joberrors: 0
 jobmissingfiles: 0
          poolid: 0
        poolname:
      priorjobid: 0
        priorjob:
       filesetid: 0
         fileset:
        hascache: 0
         comment: VolumeToCatalog of job: debian11-DR6-backup
        reviewed: 0

   232 | debian11-DR6-backup | 2023-11-04 13:40:16 | B    | I     |      446 |  1,243,777,653 | T         |
+-------+---------------------+---------------------+------+-------+----------+----------------+-----------+
*ll joblog jobid=232
    time: 2023-11-04 13:40:16
 logtext: debian11-DR6-dir JobId 232: Start Backup JobId 232, Job=debian11-DR6-backup.2023-11-04_13.40.14_24

    time: 2023-11-04 13:40:16
 logtext: debian11-DR6-dir JobId 232: Connected to Storage "File1" at debian11-DR6:9103 with TLS

    time: 2023-11-04 13:40:16
 logtext: debian11-DR6-dir JobId 232: Using Device "FileChgr1-Dev1" to write.

    time: 2023-11-04 13:40:16
 logtext: debian11-DR6-dir JobId 232: Connected to Client "debian11-DR6-fd" at debian11-DR6:9102 with TLS

    time: 2023-11-04 13:40:16
 logtext: debian11-DR6-fd JobId 232: Connected to Storage at debian11-DR6:9103 with TLS

    time: 2023-11-04 13:40:16
 logtext: debian11-DR6-dir JobId 232: Sending Accurate information to the FD.

    time: 2023-11-04 13:40:24
 logtext: debian11-DR6-sd JobId 232: Volume "Vol-0436" previously written, moving to end of data.

    time: 2023-11-04 13:40:24
 logtext: debian11-DR6-sd JobId 232: Ready to append to end of Volume "Vol-0436" size=13,900,450,583

    time: 2023-11-04 13:40:42
 logtext: debian11-DR6-sd JobId 232: Elapsed time=00:00:18, Transfer rate=69.10 M Bytes/second

    time: 2023-11-04 13:40:42
 logtext: debian11-DR6-sd JobId 232: Sending spooled attrs to the Director. Despooling 84,314 bytes ...

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: Bacula Enterprise debian11-DR6-dir 13.0.3 (02May23):
  Build OS:               x86_64-pc-linux-gnu-bacula-enterprise debian 11.2
  JobId:                  232
  Job:                    debian11-DR6-backup.2023-11-04_13.40.14_24
  Backup Level:           Incremental, since=2023-11-04 13:12:46
  Client:                 "debian11-DR6-fd" 13.0.3 (02May23) x86_64-pc-linux-gnu-bacula-enterprise,debian,11.2
  FileSet:                "Full_Set" 2023-11-02 13:36:36
  Pool:                   "File" (From Command input)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "File1" (From Command input)
  Scheduled time:         04-Nov-2023 13:40:14
  Start time:             04-Nov-2023 13:40:16
  End time:               04-Nov-2023 13:40:43
  Elapsed time:           27 secs
  Priority:               10
  FD Files Written:       446
  SD Files Written:       446
  FD Bytes Written:       1,243,777,653 (1.243 GB)
  SD Bytes Written:       1,243,836,991 (1.243 GB)
  Rate:                   46065.8 KB/s
  Software Compression:   None
  Comm Line Compression:  68.8% 3.2:1
  Snapshot/VSS:           no
  Encryption:             no
  Accurate:               yes
  Volume name(s):         Vol-0436
  Volume Session Id:      1
  Volume Session Time:<PHONE_NUMBER>
  Last Volume Bytes:      15,145,218,406 (15.14 GB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Termination:            Backup OK

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: Begin pruning Jobs older than 3 years .

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: No Jobs found to prune.

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: Begin pruning Files.

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: No Files found to prune.

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: End auto prune.

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: shell command: run AfterJob "/opt/bacula/scripts/verify.sh 232 debian11-DR6-fd debian11-DR6-backup Full_Set"

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: AfterJob: VolumeToCatalog Job's jobid=233

    time: 2023-11-04 13:40:43
 logtext: debian11-DR6-dir JobId 232: AfterJob: Queueing DiskToCatalog verify of job "debian11-DR6-backup", (jobid 232)

           jobid: 232
             job: debian11-DR6-backup.2023-11-04_13.40.14_24
            name: debian11-DR6-backup
     purgedfiles: 0
            type: B
           level: I
        clientid: 1
      clientname: debian11-DR6-fd
       jobstatus: T
       schedtime: 2023-11-04 13:40:14
       starttime: 2023-11-04 13:40:16
         endtime: 2023-11-04 13:40:43
     realendtime: 2023-11-04 13:40:43
        jobtdate: 1,699,065,643
    volsessionid: 1
  volsessiontime: 1,699,065,585
        jobfiles: 446
        jobbytes: 1,243,777,653
       readbytes: 1,243,777,653
       joberrors: 0
 jobmissingfiles: 0
          poolid: 2
        poolname: File
      priorjobid: 0
        priorjob:
       filesetid: 8
         fileset: Full_Set
        hascache: 0
         comment:
        reviewed: 0

Hey there...
Job 233 has a jobstatus of C which means it is Created, but not yet running
What does status director show?

*status director
debian11-DR6-dir Version: 13.0.3 (02 May 2023) x86_64-pc-linux-gnu-bacula-enterprise debian 11.2
Daemon started 04-Nov-23 13:39, conf reloaded 04-Nov-2023 20:31:14
 Jobs: run=0, running=3 mode=0,0
 Crypto: fips=N/A crypto=OpenSSL 1.1.1k  25 Mar 2021
 Heap: heap=176,128 smbytes=649,101 max_bytes=901,832 bufs=900 max_bufs=956
 Res: njobs=4 nclients=1 nstores=1 npools=3 ncats=1 nfsets=3 nscheds=2

Scheduled Jobs:
Level          Type     Pri  Scheduled          Job Name           Volume
===================================================================================
Full           Backup    11  05-Nov-23 23:10    BackupCatalog      Vol-0436
====

Running Jobs:
Console connected using TLS at 04-Nov-23 13:40
Console connected using TLS at 04-Nov-23 19:28
 JobId  Type Level     Files     Bytes  Name              Status
======================================================================
   232  Back Incr        446    1.243 G debian11-DR6-backup has terminated
   233  Veri Volu          0         0  Verify_Catalog    is waiting on max Client jobs
   234  Back Full          0         0  BackupCatalog     is waiting for higher priority jobs to finish
====

Terminated Jobs:
 JobId  Level     Files      Bytes   Status   Finished        Name
====================================================================
   223  Disk           0         0   Cancel   04-Nov-23 13:12 Verify_Catalog
   225  Data           0         0   Cancel   04-Nov-23 13:12 Verify_Catalog
   221  Incr       1,239    1.257 G  OK       04-Nov-23 13:12 debian11-DR6-backup
   226  Volu           0         0   Cancel   04-Nov-23 13:21 Verify_Catalog
   227  Disk           0         0   Cancel   04-Nov-23 13:21 Verify_Catalog
   228  Data           0         0   Cancel   04-Nov-23 13:21 Verify_Catalog
   224  Incr         271    878.5 M  OK       04-Nov-23 13:21 debian11-DR6-backup
   229  Volu         271         0   OK       04-Nov-23 13:24 Verify_Catalog
   230  Disk     252,272         0   Diffs    04-Nov-23 13:32 Verify_Catalog
   231  Data         271    878.5 M  OK       04-Nov-23 13:32 Verify_Catalog

====

Yes, and there it is... is waiting on max Client jobs
You will need to edit the Client resource in the Director for this client and add MaximumConcurrentJobs = X where X is greater than 1.
Additionally, you will need to check the bacula-fd.conf file on this client and also make sure it has this seeting greater than 1.
You will need to restart the bacula-fd if you make a change to its configuration, and you will need to reload the director's config after making the change to the client resource.
These changes should solve this issue for you.  \o/
I would cancel jobid=233 first, then jobid232 will finish and so will the other queued ones. Then make the changes and rerun the backup job.

Hello,
That worked. Thanks!
I'd like to schedule a copy job on successful completion of the verify, can you offer any tips on how to do that?


GSLIB plotting functions should allow for axis specification

Hi Dr. Pyrcz,
I'm currently doing your HW #2 and thought it might be useful to include axis arguments in locmap_st and hist_st so that one can supply ax1, ax2, ax3 args for further subplot customization.
I can try and create a PR sometime, but wanted to bring it up.
Jake G.

Howdy Jake,
That is a great idea. While ‘1-liners’ are useful for convenient, concise workflow code, flexibility is also important. It is a challenge to balance this! GeostatsPy was written on the weekends, days before students needed the code in my lectures! You will see many opportunities to improve.
o. I like this idea.
Remember, any improvements implemented in GeostatsPy will result in you being added to the author list with specification of your contribution!
All the best,
Michael
MICHAEL J. PYRCZ, Ph.D., P.Eng.,  Associate Professor
The University of Texas at Austin | Twitterhttps://twitter.com/GeostatsGuy | YouTubehttps://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig | GitHubhttps://github.com/GeostatsGuy | LinkedInhttps://www.linkedin.com/in/michael-pyrcz-61a648a1/ | Webpagehttp://www.michaelpyrcz.com/ | Bookhttps://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446 | GoogleScholarhttps://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en | DiReCT Consortiumhttps://direct.pge.utexas.edu/
From: Jake Gearon<EMAIL_ADDRESS>Sent: Tuesday, September 15, 2020 4:27 PM
To: GeostatsGuy/GeostatsPy<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: [GeostatsGuy/GeostatsPy] GSLIB plotting functions should allow for axis specification (#29)
Hi Dr. Pyrcz,
I'm currently doing your HW #2https://github.com/GeostatsGuy/GeostatsPy/pull/2 and thought it might be useful to include axis arguments in locmap_st and hist_st so that one can supply ax1, ax2, ax3 args for further subplot customization.
I can try and create a PR sometime, but wanted to bring it up.
Jake G.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/GeostatsGuy/GeostatsPy/issues/29, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHUO3465NMESRGZA5NZY5R3SF7LZBANCNFSM4RNZTCAA.
This message is from an external sender. Learn more about why this matters.https://ut.service-now.com/sp?id=kb_article&number=KB0011401

Pyrcz,
Sounds good, I just try my best to flag things as I use open source packages (and hopefully PR them as a follow-up when possible). I'll do the PR this weekend. See you tomorrow!
Jake

keeping it open until I do it. Cheers


Publishing api v2 endpoints

Depends on this being merged: https://github.com/alphagov/gds-api-adapters/pull/360
Jenkins won't pass until that branch is merged into master because this project depends on the PACT json file it generates. Everything is passing locally.

General comment: we prefer commit messages to have a specific summary of what's changed, rather than eg "This isn't used". See https://github.com/alphagov/styleguides/blob/master/git.md

Thanks @danielroseman - I have reworded.

looks good.
I agree it's good to standardise on symbol or string keys for hashes.
There's one important piece of functionality missing from this which is to combine the links with content when sending to draft content store. I'd suggest that we should make a separate ticket for that and go ahead and merge this pull request so that we can make progress with integrating it against #78

Agreed - we should write something up in Trello. I'll do that now.

We're going to just check this validation:
"base path may not change if this item has been previously published"
Don't merge yet.


Upgraded sklearn version to <0.24

Issue #, if available:
Description of changes:
Upgraded sklearn version to <0.24
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Job PR-671-1 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-671/1/index.html


ARROW-13033: [C++] Kernel to localize naive timestamps to a timezone (preserving clock-time)

This is to resolve ARROW-13033.

Thanks for the review @jorisvandenbossche :).
I've gone through your feedback and done some of the boilerplate except for the correct output timestamp type. I'll try to do it over the weekend.

(also, given all the confusion also on the JIRA, we should probably try to come up with a more explicit/descriptive name ...)

I think there is no consensus on this yet. tz_convert, tz_localize were mentioned. I'd add tz_normalize. Any other?

@rok can you explain the rationale of adding a source_timezone and destination_timezone to the API? What's the reason for doing it like this, instead of having a single target timezone as argument?

@rok can you explain the rationale of adding a source_timezone and destination_timezone to the API? What's the reason for doing it like this, instead of having a single target timezone as argument?

In case we start with "naive" a timestamp we don't know the source and target timezone so both need to be specified. Or am I missing something here?

What's the difference between both timezones? In my mind this is the same time zone? If you want to convert your localized timezone-aware timestamp to another timezone after localizing, you can use another kernel to do that? (a kernel we don't have yet, but one we should also add, although this is a trivial, metadata-only change)

If you want to convert your localized timezone-aware timestamp to another timezone after localizing, you can use another kernel to do that? (a kernel we don't have yet, but one we should also add, although this is a trivial, metadata-only change)

I was thinking only of the metadata change not "materialization" of a target timezone. Having another kernel to only change metadata seems redundant.

Having another kernel to only change metadata seems redundant.

I don't think that's redundant, as a user will regularly want to change the timezone of a timestamp column that is already localized (already has a timezone).
I am of course coming from the point of view of pandas, where those two operations are defined as two distinct methods: tz_localize to convert from "timestamp without timezone" to "timestamp with timezone" (i.e. from naive clock time to tz-aware time) and tz_convert to convert between "timestamp with timezone" with different timezones (a metadata-only change).
From a user point of view (and also implementation-wise), those two are distinct concepts, IMO, and so I would keep them as separate kernels. (also eg R's lubridate has distinct force_tz and with_tz, respectively)

(if we keep both source and destination timezone in the "localize" kernel, I think at least the destination timezone, if not specified, should default to be the same as the source timezone, and not "UTC", so that a user doesn't have to specify the timezone they want twice, like localize(timestamp, source_timezone="Europe/Brussels", destination_timezone="Europe/Brussels") to end up with a timestamp(unit, tz="Europe/Brussels) type column)

I opened https://issues.apache.org/jira/browse/ARROW-13247 for the discussion whether we want a separate "change timezone" kernel

Having another kernel to only change metadata seems redundant.

I don't think that's redundant, as a user will regularly want to change the timezone of a timestamp column that is already localized (already has a timezone).
I am of course coming from the point of view of pandas, where those two operations are defined as two distinct methods: tz_localize to convert from "timestamp without timezone" to "timestamp with timezone" (i.e. from naive clock time to tz-aware time) and tz_convert to convert between "timestamp with timezone" with different timezones (a metadata-only change).
From a user point of view (and also implementation-wise), those two are distinct concepts, IMO, and so I would keep them as separate kernels. (also eg R's lubridate has distinct force_tz and with_tz, respectively)

I suppose these are two different operations indeed. I've changed to make this only "localization" now.
I like force_tz as it indicates the operation is a bit "dangerous". Candidates are then:

force_tz
tz_localize
tz_normalize

Since the Arrow spec is going to refer to "LocalDateTime", "Instant" and "ZonedDateTime", how about reusing terms from the spec? Something along the lines of local_datetime_to_zoned_datetime()?

Since the Arrow spec is going to refer to "LocalDateTime", "Instant" and "ZonedDateTime", how about reusing terms from the spec? Something along the lines of local_datetime_to_zoned_datetime()?

It's nice that it's idiomatic if a bit verbose :)
Let's add it to the list. If we don't get to consensus here we can always call a vote on the ML.

force_tz
tz_localize
tz_normalize
local_datetime_to_zoned_datetime

I like all except tz_normalize.  "normalize" just has too many meanings for me already.

@rok I was checking out this PR, and in the process wrote python bindings for the option class to be able to test it in Python, so thought to directly push that here as well. (and at the same time also fixed (I think) the initialization of the timezone, which was causing the C++ tests to fail here as well)
And from testing, you will still need to handle the errors raised by date.h:
In [21]: arr = pa.array([pd.Timestamp("2021-03-28 02:30:00")], type=pa.timestamp("ns"))

In [22]: pc.localize(arr, options=pc.TemporalLocalizationOptions("Europe/Brussels"))
terminate called after throwing an instance of 'arrow_vendored::date::nonexistent_local_time'
  what():  2021-03-28 02:30:00.000000000 is in a gap between
2021-03-28 02:00:00 CET and
2021-03-28 03:00:00 CEST which are both equivalent to
2021-03-28 01:00:00 UTC
Aborted (core dumped)

@rok I was checking out this PR, and in the process wrote python bindings for the option class to be able to test it in Python, so thought to directly push that here as well. (and at the same time also fixed (I think) the initialization of the timezone, which was causing the C++ tests to fail here as well)

Thanks @jorisvandenbossche! The new option initialization was indeed problematic.

@jorisvandenbossche I've added nonexistent and ambiguous time handling. I'll try to template calls instead of having a single general one.
What I'm not sure about now is if try/catch is the way to go here and how to return NaN when the timestamp is an int64.
Another thing: infer requires understanding if the array is monotonous or not (if I understand correctly) and I'd push it out of scope of this PR for now.

What I'm not sure about now is if try/catch is the way to go here

I think the try/catch is fine, it's the way to convert an exception (which we don't use ourselves as return values) into a Status.

and how to return NaN when the timestamp is an int64.

I think it should rather be a null instead of NaN (I suppose this is for the IGNORE case?), and nulls can be encoded in int64.
I also think that the output type of the kernel should be timestamp with timezone.

Another thing: infer requires understanding if the array is monotonous or not (if I understand correctly) and I'd push it out of scope of this PR for now.

Yes, that's certainly something to be left/discussed for later, as it makes the kernel no longer a simple scalar kernel (where we can operate on each scalar independently element-wise), since the next scalar needs to have knowledge about the previous ones.

and how to return NaN when the timestamp is an int64.

I think it should rather be a null instead of NaN (I suppose this is for the IGNORE case?), and nulls can be encoded in int64.

Yes I was thinking about the IGNORE case. This should now return nulls but zeros are returned. Do we need to switch away from ScalarUnaryNotNullStateful for this?

I also think that the output type of the kernel should be timestamp with timezone.

Done. I was not sure how to get the timezone into the out type. I've made a runtime resolver now.
I'll move this out of draft to invite more discussion :).

CI fails with:
C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/i386/libarrow.a(tz.cpp.obj):(.text+0x725): undefined reference to `_imp__CoTaskMemFree@4'
C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/i386/libarrow.a(tz.cpp.obj):(.text.unlikely+0x3e): undefined reference to `_imp__CoTaskMemFree@4'

It seems to be due to tz object being problematic on windows (where currently tzdb is not available).
It appears that ARROW-13033 and ARROW-12980 are now both blocked by ARROW-13168.

I removed the R bindings for now and we can reintroduce it with ARROW-13168.

That did not help. This suggests adding -lole32 to mingw.

and how to return NaN when the timestamp is an int64.

I think it should rather be a null instead of NaN (I suppose this is for the IGNORE case?), and nulls can be encoded in int64.

Yes I was thinking about the IGNORE case. This should now return nulls but zeros are returned. Do we need to switch away from ScalarUnaryNotNullStateful for this?

If it makes the implementation more complicated (as it would no longer simply propagate nulls), I think it is fine to also leave this option for a future enhancement

and how to return NaN when the timestamp is an int64.

I think it should rather be a null instead of NaN (I suppose this is for the IGNORE case?), and nulls can be encoded in int64.

Yes I was thinking about the IGNORE case. This should now return nulls but zeros are returned. Do we need to switch away from ScalarUnaryNotNullStateful for this?

If it makes the implementation more complicated (as it would no longer simply propagate nulls), I think it is fine to also leave this option for a future enhancement

I suppose it would. Shall we leave this PR as is then?

Shall we leave this PR as is then?

I would then remove this option (if we keep it in the PR, it should do the correct thing IMO, while not it is still returning 0 (i.e. 1970-01-01) instead I think?)

Can you also handle the case of an unknown / wrong timezone? (I thought I already commented about that before, but don't directly see it ;)) Currently when using such a timezone, you get a segfault:
In [7]: pc.tz_localize(arr, timezone="Europe/Brussels", nonexistent="shift_forward").type
Out[7]: TimestampType(timestamp[ns, tz=Europe/Brussels])

In [8]: pc.tz_localize(arr, timezone="Europe/Brusselsss", nonexistent="shift_forward")
terminate called after throwing an instance of 'std::runtime_error'
  what():  Europe/Brusselsss not found in timezone database
Aborted (core dumped)

Thanks for the review @jorisvandenbossche! :)

Shall we leave this PR as is then?

I would then remove this option (if we keep it in the PR, it should do the correct thing IMO, while not it is still returning 0 (i.e. 1970-01-01) instead I think?)

Removed the IGNORE option and made ARROW-13347 to discuss/remember.

Can you also handle the case of an unknown / wrong timezone?

Added but not at TemporalLocalizationOptions initialization but in the TzLocalize exec since state can be returned from there.
I think this is ready for another review.

@pitrou - I've rebased this and it's ready for review.

We still have the naming issue (or bikeshedding) to resolve. As a reference, some names used by other systems that I am aware of:

Pandas: tz_localize
R's lubridate: force_tz
Java: atZone
Java Joda: toDateTime
Postgresql: AT TIME ZONE

The proposals that have been floated around above:

tz_localize
force_tz
local_datetime_to_zoned_datetime

I left out tz_normalize (like Weston mentioned, "normalize" already has too many meanings, it also doesn't give any indication to me whether it's adding or removing a timezone ..).
I personally like Adam's suggestion of being more explicit (although resulting in a longer name) and re-using the terminology that was now added to the format spec to clarify the meaning. I would maybe only use "timestamp" instead of "datetime", since the actual types in arrow that this kernel accepts are called that way.
With that, I would do a proposal of calling it timestamp_local_to_zoned ?

As for naming the function, perhaps "localize_timestamp"? Most important is to have a good explanatory docstring.

Although it's named like this in pandas, I am a bit hesitant to use "localize". I seem to recall that eg @bkietz interpreted that as the reverse operation (from zoned to naive/local), and in general it seemed to cause confusion in the discussion around this.
And while typing that, I realize that it would indeed be nice to have a kernel for the reverse as well (zoned to local), so the question is how to name that if we call this "localize_timestamp" (while for "timestamp(_local)_to_zoned" there is a obvious "timestamp(_zoned)_to_local" equivalent naming option)

"localize_naive_timestamp" perhaps?

The problem is we don't have any clear terminology here. "zoned" doesn't mean anything to me.

I was following the new terminology at
https://github.com/apache/arrow/blob/895f85fd32782a4447db74e4413e54a4f7128809/format/Schema.fbs#L221-L247

I don't read that as new terminology, rather pointers to terminology used in some other pieces of software.
(also I find that explanation actually a bit intimidating and confusing)

Next name suggestion: to_timestamp_with_time_zone.
It doesn't use any specific terminology like "local" or "zoned", but only refers to the actual parameter of the timestamp type: whether it has the time zone set or not.

I don't read that as new terminology, rather pointers to terminology used in some other pieces of software.
(also I find that explanation actually a bit intimidating and confusing)

The edits in https://github.com/apache/arrow/pull/10629 where meant to clarify the text after the discussion on the mailing list (and not make it even more confusing :)), so if that's not the case, I think your feedback would be welcome.

@jorisvandenbossche I think most of the names we're discussing are great. I would slightly prefer an 'old' name as it confuses slightly less users :).
I'm just going through the feedback now and will push a commit today.

@jorisvandenbossche ok, this is ready for another review.

Name proposals so far:

tz_localize
force_tz
local_datetime_to_zoned_datetime
timestamp_to_zoned
timestamp_to_local
timestamp_local_to_zoned
timestamp_local_at_zone
localize_timestamp
localize_naive_timestamp
to_timestamp_with_time_zone

I kinda like '..._at_zone' format because it makes me think the timestamp will be interpreted as in a timezone rather then converted to.

I'm going to suggest assume_timezone.

@rok Thanks for asking! I prefer assume_timezone, too.

CI fail seems unrelated.

Ok, I've pushed some minor changes and will merge once CI is green. Thanks a lot for this @rok !

Let me rebase that.

@rok I'm on it already.

Thanks @jorisvandenbossche @pitrou @adamhooper @westonpace !


Bump cdap version to 6.7.3 snapshot

[x] To be merged after cdap-build for 6.7.3-SNAPSHOT is created.
https://github.com/cdapio/cdap-build/releases/tag/v6.7.3-SNAPSHOT

Validated that scenario failing in the build is working for me with 6.7.3 sandbox on local.
Looks like a flaky test, running another build.

Please also change the version in this script - https://github.com/cdapio/cdap-e2e-tests/blob/2b21dd2b64ffe2801669968b6ec960fde74b85dc/src/main/scripts/run_e2e_test_module.py#L32

Please also change the version in this script -
https://github.com/cdapio/cdap-e2e-tests/blob/2b21dd2b64ffe2801669968b6ec960fde74b85dc/src/main/scripts/run_e2e_test_module.py#L32

Are we still using https://builds.cask.co ? OR should it be changed to github link?
Is run_e2e_test_module.py used anywhere?

Please also change the version in this script -
https://github.com/cdapio/cdap-e2e-tests/blob/2b21dd2b64ffe2801669968b6ec960fde74b85dc/src/main/scripts/run_e2e_test_module.py#L32

Are we still using https://builds.cask.co ? OR should it be changed to github link?
Is run_e2e_test_module.py used anywhere?

It should be changed to github link, not sure if this script is used anywhere.


isabelle build fails when generating a document

Describe the bug
When using Isablle to build a session with a document output, the build fails with an I/O error. The cause seems to be that some template files (e.g. lib/texinputs/isabelle.sty) are copied from the store (as read-only) and then Isabelle attempts to write to these files.
Steps To Reproduce
Steps to reproduce the behavior:

isabelle mkroot
isabelle build -D .

Expected behavior
For the isabelle build -D . invocation to not fail
Additional context
The root cause seems to be that some files are copied from the store for later modification using this copy function in the end, which copies the file as read-only from the nix-store. The build then dies when it tries to modify the read-only file.
I am not sure what the best fix here is. My first idea would be to just patch the copy function in Isabelle to allow the owner to write to the file when copying a file from the nix-store.
For reference, the output of building a freshly generated session:
$ pwd
/tmp/isabelle-example
$ isabelle mkroot

Initializing session "isabelle-example" in "/tmp/isabelle-example"
  creating "ROOT"
  creating "document/root.tex"

Now use the following command line to build the session:

  isabelle build -D .

$ isabelle build -D .
Running isabelle-example ...
Preparing isabelle-example/document ...
I/O error: /tmp/isabelle-example/output/document/isabelle.sty
isabelle-example FAILED (see also "isabelle build_log -H Error isabelle-example")
*** I/O error: /tmp/isabelle-example/output/document/isabelle.sty
Unfinished session(s): isabelle-example
0:00:02 elapsed time

Notify maintainers
@jwiegley @jvanbruegge

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.75, NixOS, 23.11 (Tapir), 23.11.20240127.56911ef`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(lv): `""`
 - nixpkgs: `/nix/store/xngp51ajwazakjpilwjqk6db4vsg65ga-source`

Add a :+1: reaction to issues you find important.

I am using this overlay as a workaround for now, that patches the copy_file function:
final: prev:

{
  isabelle = prev.isabelle.overrideAttrs (old: {
    patches = (old.patches or []) ++ [./isabelle-copy.patch];
  });
}

And the patch:
diff --git a/src/Pure/System/isabelle_system.scala b/src/Pure/System/isabelle_system.scala
index 687014f..9b211a2 100644
--- a/src/Pure/System/isabelle_system.scala
+++ b/src/Pure/System/isabelle_system.scala
@@ -215,6 +215,9 @@ object Isabelle_System {
         Files.copy(src.toPath, target.toPath,
           StandardCopyOption.COPY_ATTRIBUTES,
           StandardCopyOption.REPLACE_EXISTING)
+        if (src.toPath().toAbsolutePath().startsWith("/nix/store/")) {
+          target.setWritable(true)
+        }
       }
       catch {
         case ERROR(msg) =>

The check, if the path starts with /nix/store doesn't feel like a good way to check if we are copying from the nix-store, but I am not sure what the "correct" way to do this would be.

I think removing the check alltogether would be fine too. The target file should always be writeable

Thanks for the suggestion. I did that for the patch in the PR


Migrate SB and Unified Build Installer Pipelines to SDK

As part of the work to merge Installer to SDK, the SB pipelines will need to be updated to be run out of SDK. This work should be completed concurrently with the work to update the synchronization (@ViktorHofer).
This pipeline work is broken into a few steps:

[x] Copy over installer/eng/pipelines into sdk/eng/pipelines: https://github.com/dotnet/sdk/pull/40420
[x] Modify ci.yml to use sdk/eng/pipelines templates instead of installer templates: https://github.com/dotnet/sdk/pull/40451
[x] Create duplicate of the internal installer-source-build pipeline for sdk: NEW PIPELINE (internal Microsoft link)
[x] Create duplicates of the PR validation pipelines

[ ] The rules will need to be changed to match those of installer in terms of using the new SDK pipelines https://github.com/dotnet/installer/settings/branch_protection_rules/1970909 - Waiting on green SB and UB builds out of SDK.

[ ] Create a shadow of the SB release pipeline that releases from the SDK instead of installer. Once this is working and we've deprecated main in installer, the actual SB release pipeline should be updated to match this "shadow".

DRAFT PR (internal Microsoft link).
Waiting to publish until Staging changes go in, but otherwise changes in the draft PR are ready.

[x] Update the SB monitor dashboard to include main-sdk branch

Current progress:

dotnet-source-build-sdk pipeline configured, based on my internal Azdo dev branch
dotnet-source-build-lite-sdk pipeline configured, based on my internal Azdo dev branch

AFAIK the only thing remaining is

Create a shadow of the SB release pipeline that releases from the SDK instead of installer. Once this is working and we've deprecated main in installer, the actual SB release pipeline should be updated to match this "shadow".

The sync now works and we will soon hopefully have green SB and unified-build PR builds out of sdk.

@ellahathaway don't we need to keep the installer-based release pipeline around? For 8.0 releases? Possibly even for 7.0 and 6.0?

don't we need to keep the installer-based release pipeline around? For 8.0 releases? Possibly even for 7.0 and 6.0?

Absolutely! The current infra of the source-build release pipeline allows for a simple change to be made in order to use of the sdk-based VMR instead of the installer-based VMR. This change is localized to 9.0, meaning that the pipeline will still support installer-based releases for <= 8.0.

Last remaining step is blocked on the staging pipeline work.

The PR has gone in, so I'm closing this issue.


Usability improvements proposal

Add editable keyboard shortcuts to quik set/remove highlights. For example Ctrl+1 - fuchsia Ctrl+2 - gold, etc.
Add context menu option to remove signle highlight from document (where cursor is), not all.
Add mark on the vertical scroll bar for quik navigation. May be it is better to draw it is as a horizontal line with appropriate color, to not confuse with many dots that may be shown by VS itself.
Editable colors

Add mark on the vertical scroll bar for quik navigation. May be it is better to draw it is as a horizontal line with appropriate color, to not confuse with many dots that may be shown by VS itself.

Will investigate.

There doesn't seem to be an API that enables adding to the annotations that VS shows in the scroll bar :(

There doesn't seem to be an API that enables adding to the annotations that VS shows in the scroll bar :(

Here is an attempt to draw on VS scrollbar in similar type of extension: oleg-shilo/PyMap.VSIX@a2389fc

Thanks for that, it helped me find a solution that is in v1.4


change order of parameters for Im2col function

For convolution.cc, change the order of parameters for Im2col and Col2im functions.

xiangrui, pls follow this guide  http://singa.apache.org/develop/contribute-code.html to contribute code to SINGA. Thanks.

OK, thanks. I'll do it.
Best regards!
Xiangrui
On Fri, Jan 15, 2016 at 4:42 PM, Wei Wang<EMAIL_ADDRESS>wrote:

xiangrui, pls follow this guide
http://singa.apache.org/develop/contribute-code.html to contribute code
to SINGA. Thanks.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-singa/pull/120#issuecomment-171903011
.

have you updated the commit msg title?

Pls close this pull request as it is resolved in #126


Add a TapirMode that we can pass to AutoTapir(mode=TapirMode(debug=true))

It will provide flexible control over how Tapir is run but avoid breaking changes to ADTypes. If TapirMode is changed in a breaking manner, we simply mark the corresponding Tapir release breaking so users won't be surprised.
The name TapirMode can be anything we agree on.

This is a good idea. We will need to do this prior to renaming / updating ADTypes.jl with the a new type.

Resolved by #262


[Feature Request] Add dir attribute for each locale

Is your feature request related to a problem? Please describe.
I'm using this package to translate my app, and now I'm adding an RTL locale.
It would be great if nuxt-i18n could manage the body dir attribute.
Describe the solution you'd like
I propose to add a dir property for each locale in the module (values would be rtl, ltr or auto by default). For example:
// nuxt.config.js, inside nuxt-i18n module

locales: [
  { code: 'ar', iso: 'ar', file: 'ar/app.js', dir: 'rtl' },
  { code: 'en', iso: 'en-US', file: 'en/app.js', dir: 'ltr' },
  { code: 'fr', iso: 'fr-FR', file: 'fr/app.js', dir: 'ltr' },
],

Then this would be added in the HTML by setting a new dir attribute at a very top level of the document, for example in the  or  tag. I think it's a mandatory feature for an i18n repo 😊
Describe alternatives you've considered
As for now I'm using a custom plugin and adding the dir property as soon as I can:
// plugins/ltr-rtl.js

export default function({ app }, inject) {
  const dir = () => app.i18n.locales.find((x) => x.code === app.i18n.locale)?.dir;
  inject( 'dir', dir);
}

  My app here...

What do you think? 😊

RTL always has so little support, so I'm happy to see it being brought up!
I have a similar issue that I'm trying to resolve. I tried two different solutions, and both are unsatisfactory. I'm still fairly new to Nuxt, so I'm not 100% sure this is the right approach. It would be good to create a more "standard" way of solving the rtl problem.
export default function({ app }) {
  // beforeLanguageSwitch called right before setting a new locale
  app.i18n.beforeLanguageSwitch = (oldLocale, newLocale) => {
    app.head.htmlAttrs = {
      lang: newLocale,
      dir: newLocale == "ar" ? "rtl" : "ltr"
    }
  }
}

Nevertheless, it would be great to have a $dir property to use where needed.

Hey @rch, I would really love to have your opinion 😊

Great idea and should be easy to do. :)

You can also target the direction CSS property as a workaround - https://developer.mozilla.org/en-US/docs/Web/CSS/direction
e.g.
html[lang="ar-SA"] {
  direction: rtl;
}

@glenpike Using the HTML attribute is way much better than the CSS property 😊

@glenpike Using the HTML attribute is way much better than the CSS property 😊

Hi, yes, sorry, should have commented this was a workaround.

Hi @rchl, any news on this feature request? 😊

Is this already implemented?

We can leverage from the SEO mode when enabled,  it already adds the ISO to the HTML so it will be so easy to do add the dir, or you intend to make it a stand-alone feature?

We can leverage from the SEO mode when enabled,  it already adds the ISO to the HTML so it will be so easy to do add the dir, or you intend to make it a stand-alone feature?

Hi,  i'm trying to make this done, as i notice, we need to combine both the meta which coming when the seo-mode is enabled and the dir attribute that will be coming from the dir when be implemented - in the httmlattrs, is it the best approach? any ideas?

Hi,  i'm trying to make this done, as i notice, we need to combine both the meta which coming when the seo-mode is enabled and the dir attribute that will be coming from the dir when be implemented - in the httmlattrs, is it the best approach? any ideas?

Resolved through #1023 and released in https://github.com/nuxt-community/i18n-module/releases/tag/v6.19.0.
See the dir property in https://i18n.nuxtjs.org/options-reference#locales

@rchl How can we get that property in pages or components?

@aliwesome not sure what you mean exactly but you need to add this in the default layout or individual pages:
export default {
  head () {
    return this.$nuxtI18nHead()
  }
}

I made this solution and worked fine:
First, create 'localize.js' as a plugin file in the plugins folder and add this code :
export default ({ app }) => {
  app.router.afterEach((to, from) => {
    if (app.i18n.locale === 'ar') {
      app.head.htmlAttrs.dir = 'rtl'
      app.head.htmlAttrs.lang = 'ar'
    } else {
      app.head.htmlAttrs.dir = 'ltr'
      app.head.htmlAttrs.lang = 'en'
    }
  })
}

This file will be imported to the nuxt.config.js file :
plugins: [
    '~/plugins/localize.js',
  ],

Then, Go to your nav component file and add this head function in your script to handle route changes:
head() {
    if (this.$i18n.locale === 'ar') {
      return {
        htmlAttrs: {
          lang: 'ar',
          dir: 'rtl',
        },
      }
    } else {
      return {
        htmlAttrs: {
          lang: 'en',
          dir: 'ltr',
        },
      }
    }
  },

it's now supported out of the box.
https://i18n.nuxtjs.org/options-reference/#locales
Ref: https://github.com/nuxt-community/i18n-module/pull/1023

@Mo7ammedMamdou7 it's now supported out of the box. https://i18n.nuxtjs.org/options-reference/#locales Ref: #1023

Yes, bro thanks a lot for your support,
This is the configuration I made and worked fine:
First, I add this code to the i18n config in the nuxt.config.js file:
locales: [
      {
        code: 'ar',
        name: 'Arabic',
        iso: 'ar-UAE',
        file: 'ar.js',
        dir: 'rtl',
      },
      {
        code: 'en',
        name: 'English',
        iso: 'en-US',
        file: 'en.js',
        dir: 'ltr',
      },
    ],

Then I added this code to default.vue layout file:
head() {
    return this.$nuxtI18nHead({ addDirAttribute: true, addSeoAttributes: true })
  },

I wrote it on nuxt3(vue3) but I can't switch between dir for each locale
How I should fix it?
my default.vue layout file :
const head = useLocaleHead({
  addDirAttribute: true,
  addSeoAttributes: true
})

useHead({
  htmlAttrs: {
    lang: head.value.htmlAttrs!.lang,
    dir: head.value.htmlAttrs!.dir
  },
})```
my i18n config in nuxt.cofing.ts file:

i18n: {
    locales: [
        {
            code: 'fa',
            iso: 'fa-IR',
            name: 'Farsi',
            file: 'fa-IR.json',
            dir: 'rtl',
        },
        {
            code: 'en',
            iso: 'en-US',
            name: 'English',
            file: 'en-US.json',
            dir: 'ltr',
        },
    ],
    defaultLocale: 'fa',
    detectBrowserLanguage: false,
    langDir: "lang",
     // & add `vueI18n` option to `@nuxtjs/i18n` module options
    vueI18n: {
        legacy: false,
        fallbackLocale: 'fa',
    }
}

useLocaleHead

you need to add this to your layout
your template


Shouldn't be normal tap on avatar for mention and long tap for menu, and not vice versa, as we have now?

Mentions is used more frequently, than menu.

We have this design/behaviour by now on mobile clients.
We need to discuss about it.

Normal tap for mention is used already on nicknames, btw.

Current behavior is:

tap on avatar: open member detail
tap on username: mention
long tap on message: open menu

So I think this issue can be closed


Improve management endpoint mapping

MapActuatorEndpoint is now idempotent
MapAllActuators only maps endpoints that have been registered in IServiceCollection

Related to #545

/azp run

/azp run

I'm struggling to work out the code coverage issue. When I run the tests with Jetbrains dotCover, code is marked as covered.
In this file https://codecov.io/gh/SteeltoeOSS/Steeltoe/pull/560/src/src/Management/src/EndpointCore/CloudFoundry/EndpointServiceCollectionExtensions.cs
the section that is marked in red is reported to have been covered by 25 tests.

I'm struggling to work out the code coverage issue. When I run the tests with Jetbrains dotCover, code is marked as covered.
In this file https://codecov.io/gh/SteeltoeOSS/Steeltoe/pull/560/src/src/Management/src/EndpointCore/CloudFoundry/EndpointServiceCollectionExtensions.cs
the section that is marked in red is reported to have been covered by 25 tests.

@kimbell the coverage reports unfortunately don't come out 100% accurate all the time on codecov. in this case there may have been further complications due to the test failures on Mac environments... Let's revisit tests in #554

@kimbell the coverage reports unfortunately don't come out 100% accurate all the time on codecov. in this case there may have been further complications due to the test failures on Mac environments... Let's revisit tests in #554


[IA-4006] Add types for apps

This PR does the following:

adds types for Apps to conform to Leo API return
Cleans up tool-utils.ts. The concept of a tool is used throughout the code base as a non-cloud-specific application with certain display implications. Really, all a tool is is a label that uniquely identifies it regardless of the underlying cloud implementation. The model in its current state approaches this implementation, but with a layer of indirection. Essentially, app related tools had this nasty habit of having both a label and an appType that were exactly the same for properly-formed tools and not for inconsistent implementations. This is resolved and simplified by making the label for app related tools have a one-to-one mapping with the appType field returned from leonardo. This is a bit of a verbose description, so happy to chat further about this change since it affects multiple teams.
Create test file for app-specific data transformations

An example of the code smell that arose prior to 2  is the tool CromwellOnAzure. This is not really a valid tool, Cromwell is the tool, and it can exist on multiple cloud platforms. I will have a follow-up PR to detangle this overloading, but this PR is noisy as is.

@jdcanas It seems the Settings option is now being displayed for Cromwell on Azure apps. It needs to stay hidden. Maybe the isSettingsUnsupported is not being correctly checked as expected? Also the tooltip for Open has Cromwell in uppercase letters. Can they be made camel case?
Cromwell on Azure modal from dev branch:

Cromwell on Azure modal from this branch:

@jdcanas It seems the Settings option is now being displayed for Cromwell on Azure apps. It needs to stay hidden. Maybe the isSettingsUnsupported is not being correctly checked as expected? Also the tooltip for Open has Cromwell in uppercase letters. Can they be made camel case? Cromwell on Azure modal from dev branch:
Cromwell on Azure modal from this branch:

Try now @salonishah11? Not sure how it happened, but I refactored to make the logic a bit more robust.

@jdcanas It seems that now the Settings option has been hidden for all apps on both Google and Azure workspaces
Modal in Google workspace:

Modal in Azure workspace:

It also seems that clicking on the Open button in Cromwell app no longer opens the CBAS UI in new tab even though the app is running fine. Clicking on Open just dismisses the modal. Manually pasting the cbas-ui proxy url (from network tab) in new tab does load the CBAS UI. Can you verify if you are seeing this behavior too?


Merge pull request #3 from Create-With-Code-Master/master

The current branch (step -1) is already published to GitHub,

I don't think you meant to do this...


MaterialSegmentedControl frame stay to CGRect.zero

I have created a view in the storyboard and I'm trying to put the MaterialSegmentedControl in this view but it keeps the frame CGRect.zero and I can not force it to be my view frame.
let segmentedControl = MaterialSegmentedControl(segments: segments, selectorStyle: .line, textColor: .darkGray, selectorTextColor: .blue, selectorColor: .white, bgColor: .white)

// segmentedControl.frame = self.topView.frame

self.topView.addSubview(segmentedControl)

I would recommend you use AutoLayout.
Step 1. Add this line after adding the view to topView
segmentedControl.translatesAutoresizingMaskIntoConstraints = false

Step 2. In your view controller, add the following code.
override func viewDidLayoutSubviews() {
    super.viewDidLayoutSubviews()
    segmentedControl.leftAnchor.constraint(equalTo: topView.leftAnchor).isActive = true
    segmentedControl.rightAnchor.constraint(equalTo: topView.rightAnchor).isActive = true
    segmentedControl.topAnchor.constraint(equalTo: topView.topAnchor).isActive = true
    segmentedControl.bottomAnchor.constraint(equalTo: topView.bottomAnchor).isActive = true
}

This way, the frame of the segmentedControl should be forced to align with your view's frame. Let me know if this does not work.

I am using AutoLayout thanks.
I have the same crash in LayoutSubview first line:
selector.bottomAnchor.constraint(equalTo: self.bottomAnchor).isActive = true
Expected found nil...

Well this sounds like you a bug you have to figure out yourself. What does self refer to there? There might be a problem if it is not an UIView.

self refers to the UIViewController. TopView is an IBOutlet linked with the storyboard:
@IBOutlet var topView: UIView!

Right then you shouldn't use self there, UIViewController is not UIView. Use self.view or topView instead.

self.topView or topView refers to the same object. i just tried with topView and have the same issue

It's a bug you would have to fix there. I don't think it's related to segmented control itself though.

Well the issue is happening in the layoutSubviews function of the MaterialSegmentedControl class, that's why I think it is related to this. Do you have a sample of a full implementation of the Segmented control? Cause the sample I found or the nice tutorial on medium are not up to date with the framework

Try this:
let segmentedControl = MaterialSegmentedControl(selectorStyle: .line, textColor: .black, selectorTextColor: .black, selectorColor: .black, bgColor: .white)

for i in 0..<3 {
    let button = MaterialButton(text: "Segment \(i)", textColor: .gray, bgColor: .clear, cornerRadius: 12.0)
    button.rippleLayerAlpha = 0.15
    segmentedControl.segments.append(button)
}

Then add it to your whatever parent view and deal with its layout constraints. There is a MaterialDesignWidgetsDemo included in this repo, you may refer to that.

Well I could make it work by using a stackview like in the demo but still cannot use a view with constraints to put the segmented control. I think there are conflicts with the constraints.
Can you try with a normal UIView instead of StackView just to validate?

Right, I did notice that. The crash was because the segments and selector are nil. I will roll out an update to address this.


virus_table.txt

It seems to me that sometimes the virus_table.txt was not generated; in my case, it is generated when I use a small file say 100 reads, but not when I use a larger fa file say with million reads. I wonder why.

Did you work out what was happening?


chore(styles): remove temp expressive

Related Ticket(s)
Refs #3126.
Description
This change removes classic expressive theme proxy code that was used for several components.
Changelog
Changed

Replaced usage of classic expressive theme proxy code with the latest one.

Removed

Class expressive theme proxy code.

Looks good @asudoh ! I noticed in percy the button icon is moved over slightly, but according to https://carbon-expressive.mybluemix.net that appears to be the correct spacing. Might need a designer to take a closer look though? @oliviaflory

Also in web components, link with icon appears to be productive.


Javadocs fail to build on JDK 17

Expected Behavior
gradle build
should run without errors.
Actual Behavior
The following errors are reported from the javadoc task:
> Task :richtextfx:javadoc
warning: URL http://docs.oracle.com/javase/8/docs/api/element-list was redirected to https://docs.oracle.com/en/java/javase/19/ -- Update the command-line options to suppress this warning.
warning: URL http://docs.oracle.com/javase/8/javafx/api/element-list was redirected to https://docs.oracle.com/en/java/javase/19/ -- Update the command-line options to suppress this warning.
/Users/scott/dev/opensource/RichTextFX/richtextfx/src/main/java/org/fxmisc/richtext/GenericStyledArea.java:104: error: heading used out of sequence:
, compared to implicit preceding heading:

 *
Adding Scrollbars to the Area

   ^
/Users/scott/dev/opensource/RichTextFX/richtextfx/src/main/java/org/fxmisc/richtext/MultiChangeBuilder.java:17: error: heading used out of sequence:
, compared to implicit preceding heading:

 *
Relative vs Absolute Changes

   ^
/Users/scott/dev/opensource/RichTextFX/richtextfx/src/main/java/org/fxmisc/richtext/TextEditingArea.java:440: error: unexpected heading used:
, compared to implicit preceding heading:

     *
Warning: Off-By-One errors can easily occur

       ^
/Users/scott/dev/opensource/RichTextFX/richtextfx/src/main/java/org/fxmisc/richtext/model/StyledDocument.java:172: error: unexpected heading used:
, compared to implicit preceding heading:

     *
Warning: Off-By-One errors can easily occur

       ^
/Users/scott/dev/opensource/RichTextFX/richtextfx/src/main/java/org/fxmisc/richtext/StyledTextField.java:218: error: tag not supported in HTML5: tt
     * The prompt text to display or <tt>null</tt> if no prompt text is to be displayed.
                                     ^
5 errors

Environment info:

RichTextFX Version: commit 8982fd0dc7b7a13f076ca5696f3337ee9e6d5e15
Operating System: macOS 13.1
Java version: 17.0.5

Have submitted PR to address .....

I commented on the PR, it does not fix the problem, but I provided a change that does.

My bad for misreading the reported errors (didn't see the H1 - H3 conflicts, only the h3 - H3 ones)
and not actually testing against JDK 17.


broken css

On importing the UI shell. the class .bx--content has the wrong colour when comparing to the docs website.
The css with this class was generated from the latest version carbon components css.

Could you provide some more detail?

This could be a bit misguiding for someone using carbon svelte for a first time as the docs website is the main reference for importing and using components.


chore(examples): revert back to stable TS

The new router once required TS1.9 but it doesn't anymore. It's best to revert our examples to TS 1.8 since that is what we tell users to install, and we want to be testing on the current setup.
@brandonroberts @Foxandxss @wardbell

Yes please, I was getting some weird behaviors with this.

merged and live!


extend  last_spdm_error  to generic requester lib usage

Current requester_lib caller can only get libspdm defined error message. It cannot get what the responder returned error code.
We need enable libspdm_get_last_spdm_error_struct() for requester_lib as well, to allow the consumer of requester_lib can also know what error is returned from responder.
This will be used for conformance test in the future.

I will check it.

It is not needed after I review the code again. Close it.


Error when access: sh: 1: lessc: not found

I installed phundament with newest version. But when I access, error apeared like this:
AssetConverter command 'lessc '/vagrant/mysite/cms/web/assets/2f9dc4d1/less/app.less' '/vagrant/mysite/cms/web/assets/2f9dc4d1/less/app.css' --no-color --source-map' failed with exit code 127:
STDOUT:

STDERR:
sh: 1: lessc: not found

I don't know how to fix it? Any idea for this?

What OS do you use?

In general, I'd recommend using docker, since all pieces are in place there (less was a main reason for us to switch to the dockerized approach btw).
But if can or do not want to use docker, you can also install nodejs/npm to your Vagrant VM and then less.
A workaround would be to "downgrade" to CSS files, you could pick them from:

https://github.com/phundament/app/blob/4.0.0-rc8/src/assets/AppAsset.php#L22
https://github.com/phundament/app/blob/4.0.0-rc8/src/assets/web/css/app.css

I fixed this error. B/c my VM (Ubuntu Trusty) didn't installed lessc package.
I've used docker, but have some problems that I don't know. So I moved to use composer.
I will use docker again and will post errors if it happen.
Thanks all.

I've used docker, but have some problems that I don't know.

@trexanhvn Would be really interested in your problems and error messages, if you've the time create new issues, thank you!


Allow to set Environment Variables in Run Configurations

Thanks! Great suggestion, done!

Thanks! Are you planning to release a new version soon?

@miszczu yes, just need to fix few more bugs. I expect it to be ready this week.


[BACKLOG-9893] Repository Manager dialog is blank - not able to creat…

…e any repository
@bmorrise @mdamour1976

Build Completed
:white_check_mark: This pull request has passed all validations.
Build Commands
mvn -Dpentaho.resolve.repo=http://nexus.pentaho.org/content/groups/approved -B -fn -DrunITs=true -f plugins/repositories-plugin/pom.xml clean install && mvn -Dpentaho.resolve.repo=http://nexus.pentaho.org/content/groups/approved -B -f plugins/repositories-plugin/pom.xml site

Cleanup Commands
mvn -B -f plugins/repositories-plugin/pom.xml build-helper:remove-project-artifact

Changed files
  plugins/repositories-plugin/src/main/resources/META-INF/js/require.json
  plugins/repositories-plugin/src/main/resources/web/js/app.js
  plugins/repositories-plugin/src/main/resources/web/js/ui-bootstrap-tpls.js


SchNetPack/PyTorch Dependency Conflicts

Hi,
I'm experiencing some dependency conflicts and the installation instructions for the repository might need updating. schnetpack 2.0.0 requires pytorch-lightning>=1.8.1 but pytorch_lightning.loggers.base.LightningLoggerBase has been deprecated since v1.7.0. This results in the following ImportError: cannot import name 'LightningLoggerBase' from 'pytorch_lightning.loggers'. I've followed the provided installation instructions so the error should be reproducible. Any advice would be very appreciated!
Many thanks, Shay

Hi Shay,
thanks for pointing that out! The error occurs in schnetpack due to changes in the API of pytorch-lightning.
You have several options to circumvent this problem:

Install pytorch-lightning=1.8.1 manually before you install schnetpack-gschnet (since LightningLoggerBase was only removed in later versions).
Install schnetpack from source (i.e. clone the repo and install with pip install ./schnetpack) before you install schnetpack-gschnet (since the problem is already fixed in the most recent version of schnetpack)

Also, we will update the pip release of schnetpack as soon as possible. I will update this issue accordingly.

Perfect, yup that seems to have resolved the issue. Thanks Niklas!

Finally closing this issue as SchNetPack has been updated to version 2.0.1 lately and the pip-wheel should not cause conflicts anymore.