Rendering overhaul: HazeNode (was SkyBandsNode)

This PR:

refactors the SkyBandsNode class into an HazeNode class inheriting from a  generic BlurNode class.
refactors the NodeFactory class to be more flexible.
removes a number of redundant OpenGL-related directives.
Adds some JavaDocs.

This PR does not address newly opened issue #2528.
How to test

Create/Load a game and toggle on/off "fog" in the VideoSettings to see the difference with/without haze.

Haze  is an atmospheric phenomena blending faraway objects into the color of the atmosphere, or, in Terasology's case, into the color of the sky. Terasology's approach is not completely realistic (it doesn't take in account the shadows cast by faraway mountains for example) but it generally does the trick.

Hooray Jenkins reported success with all tests good!

Hooray for comments and incremental refactoring of all the rendering things!
Tested it out and couldn't find any obvious issues. The fog worked as advertised.
Am happy with merging whenever, poking @tdgunes and @indianajohn in case either is around and available for a quick comment or two :-)

Just push a commit with a tiny change. @Cervator, I imagine you'll read this on your Sunday evening. If @tdgunes hasn't offered any comment by then let's merge it.

Hooray Jenkins reported success with all tests good!

Happy to merge whenever, or you can try it out yourself :+1:

:tada:


Plugin + Jack/Java8

Hey,
actually the plugins fails to run with Jack enabled.
I get the following error with plugin and Jack enabled:
Error:Cannot cast object 'task ':app:transformJackWithJackForDebug'' with class'com.android.build.gradle.internal.pipeline.TransformTask_Decorated' to class 'org.gradle.api.tasks.SourceTask'
This might be related to #113

Closing as a duplicate of #113


Patch Tuesday June 2023 + Ubuntu 22.04

Azure Pipelines fixed Ubuntu 22.04 so we can finally adopt it.
Related:
braft https://github.com/microsoft/vcpkg/pull/31965
chipmunk https://github.com/microsoft/vcpkg/pull/31966
dbow3 https://github.com/microsoft/vcpkg/pull/31967
gppanel https://github.com/microsoft/vcpkg/pull/31968
hareflow https://github.com/microsoft/vcpkg/pull/31969
libbf https://github.com/microsoft/vcpkg/pull/31970
live555 https://github.com/microsoft/vcpkg/pull/31972
mathgl https://github.com/microsoft/vcpkg/pull/31976
mdl-sdk: https://github.com/microsoft/vcpkg/pull/32008
minisat-master-keying https://github.com/microsoft/vcpkg/pull/31977
nifly https://github.com/microsoft/vcpkg/pull/31998
sltbench https://github.com/microsoft/vcpkg/pull/31999
thor https://github.com/microsoft/vcpkg/pull/32001
unicorn-lib https://github.com/microsoft/vcpkg/pull/32002
ci.baseline.txt changes:
Need Python2
blitz
duktape
keystone
bond was just reordered together.
coin-or-ipopt fails to find lapack despite depending on intel-mkl; previously blocked by coinutils failing.
Have dynamic exception specs that are now rejected:
libodb
log4cpp
wpilib triggers what looks like a strange gcc-11 bug? Upstream has diverged substantially and it might make sense to remove the port entirely.

dcmtk matches last baseline run.
openmvg was previously blocked by coin stuff:
Building openmvg[core,software]:x64-linux...
error: building openmvg:x64-linux failed with: CASCADED_DUE_TO_MISSING_DEPENDENCIES
    due to the following missing dependencies:
        coin-or-clp[core]:x64-linux
        coin-or-osi[core]:x64-linux
        coinutils[core]:x64-linux
Elapsed time to handle openmvg:x64-linux: 153 us


Still a lot of baselines missing

the issue
I have a few Exchange 2013 and 2016 installations that still miss some baselines to compare, resulting in tons of "NoHashMatch" lines
the versions missing
[2013]
15.0.712.23
15.0.995.28
15.0.995.32
15.0.1236.6
15.0.1347.0
15.0.1347.4
[2016]
15.1.1531.4
15.1.1531.6
15.1.1531.7
15.1.1591.8
15.1.1591.16
15.1.1591.17
I have hacked together a powershell script that spits out the missing baselines, may be usefull to others (must "run as" administrator)
function FindInstalledVersions() {
    $VDIR_PATTERN = New-Object System.Text.RegularExpressions.Regex -ArgumentList  '(.+)\s+\"(.+)\"\s+\(physicalPath:(.+)\)'

    $versions = @{}

    Add-PSSnapin -Name "Microsoft.Exchange.Management.PowerShell.E2010" -ErrorAction SilentlyContinue
    $server = (Get-ExchangeServer) | Where-Object { $_.Identity.Name -eq (hostname) }
    if ($server.AdminDisplayVersion.Major -eq 14) {
        $exchange_version = (Get-Command ExSetup | ForEach-Object { $_.FileVersionInfo }).ProductVersion
    } else {
        $exchange_version = "$($server.AdminDisplayVersion.Major).$($server.AdminDisplayVersion.Minor).$($server.AdminDisplayVersion.Build).$($server.AdminDisplayVersion.Revision)"
    }
    Remove-PSSnapin -Name "Microsoft.Exchange.Management.PowerShell.E2010" -ErrorAction SilentlyContinue

    $versions[$exchange_version] = $true
    $vdir_paths = @()
    $logs = & (Join-Path $env:Windir "system32\inetsrv\appcmd.exe") LIST VDIRS | Sort-Object

    foreach ($log in $logs) {
        $log -match $VDIR_PATTERN | Out-Null;
        $vdir_physical_path = $Matches[3]
        $vdir_physical_path = $vdir_physical_path -replace "%SystemDrive%", $env:SystemDrive
        $vdir_physical_path = $vdir_physical_path -replace "%windir%", $env:windir
        $vdir_paths += $vdir_physical_path
    }

    $vdir_paths | Where-Object { Test-Path $_ } | ForEach-Object { Get-ChildItem -Directory -Path $_ } | Where-Object { $_.Name -match '^[0-9\.]*$' } | ForEach-Object { $versions[$_.Name] = $true }

    return ($versions.Keys|%{[System.Version]$_}|sort|%{$_.ToString()})
}

function CheckOnGitHub($installed_versions) {
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
    $missing=@()
    $installed_versions | ForEach-Object {
        try {
            $version=$_
            $dummy=Invoke-WebRequest -UseBasicParsing -Uri "https://github.com/microsoft/CSS-Exchange/blob/main/Security/src/Baselines/baseline_${version}.csv"
        } catch {
            $missing += $version
        }
    }
    if ($missing.Count -eq 0) {
      "All baselines are available on github"
    } else {
      "The following baselines are missing on github for your installation"
      $missing
    }
}

CheckOnGitHub(FindInstalledVersions)

edited script 11 march 2021: baselines moved to src dir on github
edited 11 march 2021: strikeout baselines now available.

Thank you for the effort you put into all this.

If this should help getting systems fixed there should really a baseline for every version hardcoded in the script.
Otherwise all the hard work is rendered useless.

Created a single task for tracking all missing baselines, please follow: https://github.com/microsoft/CSS-Exchange/issues/313


More content in event list

closes #537
Picture:

@clav1d there is some issue.

@clav1d there is some issue.

@hjiang2 as we discussed, can you fix the GUI alignment, please?

@hjiang2 as we discussed, can you fix the GUI alignment, please?

move it to draft waiting for @hjiang2 update on GUI.


New API for NodeJS SDK

Created by: Mateusz Grzywa Sep 15 2021 at 08:54 CET
Updated: Jul 08 2022 at 23:27 CET
ClickUp ID: IK-1516
I was talking with   about getting/patching DT and he said there will be new api for SDK soon. Is there any estimate on that?

Commented by: @bayegan Sep 15 2021 at 22:39 CET
@ Mateusz Grzywa : what's the difference between the API you are requesting and patchProperties
or patchPropertiesByToken APIs that currently exists? cc @yetizer

Commented by: @bayegan Sep 15 2021 at 22:39 CET
@ Stein Myrseth @ Mateusz Grzywa : ETA: QA'21

Commented by: @bayegan Sep 22 2021 at 18:25 CET
My previous comments are referring to another feature. We've delivered this feature in v0.1.7


Feature/352 dark mode

Description
Fixes #352
Type of change
Adds dark mode support, accepts 2 modes Auto or Toggle, auto will follow the browser setting and toggle will show a switch

Coverage increased (+0.1%) to 73.874% when pulling cdba7e8769164adc57fb0fd898f7a6b87a3ed326 on feature/352-dark-mode into 310f48eaf479bbe90916ea75bc02d5c75f9884f0 on develop.


Using formatting directly, and not have to apply it afterwards

In the current version it is needed to first write the text, then select it, then for instance apply (click) Bold.
It would be great and often more intuitive if it was possible to first select (click) Bold and then start writing the text.

Not possible due to some browsers content editable bugs.


Change flake8 line max to 120

Per team discussion, increasing line length tolerance to 120 from 100

I'm being patient with Jenkins :P

eh, jenkins isn't doing anything useful with this change

Phu and I just reordered the steps to make flake happen before 4-hour unit testing, though merging or not won't affect that, you're right. Just wanted to see linting run then merge I suppose.

waiting is fine too

patience is a virtue?

Ok let's all keep our eye on the "new feature branch auto checks" and "opening a PR jenkins checks" if you fail flake, you should exit with a warning within the first half hour (TIL: flake8 takes like 20 minutes for our code!) otherwise, it moves onto the lengthy unit testing...


Default Ctrl+R Ctrl+R chord handling

VSCode Version: Code 1.15.1 (41abd21afdf7424c89319ee7cb0445cc6f376959, 2017-08-16T18:05:38.342Z)
OS Version: Windows_NT ia32 10.0.15063
Extensions:

Extension
Author (truncated)
Version

vscode-eslint
dba
1.2.11

python
don
0.7.0

EditorConfig
Edi
0.9.4

PowerShell
ms-
1.4.1

Ruby
reb
0.15.0

Steps to Reproduce:

Open a JS file or any code file with editorHasRenameProvider.  Make a variable and put the cursor inside it.
Press and hold the Ctrl key.
Press R twice.
Release the Ctrl key.

Reproduces without extensions: Yes
Expected behavior: The variable will be renamed.  (Ctrl+R Ctrl+R is the default keyboard chord for renaming a variable in Visual Studio).
Actual behavior: VS Code will close the current project and open up some other project depending on what other things you've recently opened.
I wonder if there is a way to detect the chord and process the rename action.  Ctrl+R Ctrl+R is a keyboard shortcut that has been reinforced through years of muscle memory from using full Visual Studio.  Adding { "key": "ctrl+r ctrl+r",                "command": "editor.action.rename",         "when": "editorHasRenameProvider && editorTextFocus && !editorReadonly" } appears to make stand-alone Ctrl+R not work because VS Code will be waiting for the second chord.  I am willing to forgo this, but I am wondering if there is a way to support the rename action and recent files.  Sorry for the "not a bug" ticket, but it's just very frustrating to keep getting thrown out of my project when I'm just trying to do some refactoring.  :-(

I can see the pain point when you expect ctrl+r ctrl+r to work as usual. May I ask what's your expected behavior of a single ctrl+r (whether there is another ctrl+r coming or not)? Should the editor wait for the next keystroke or simply apply what's matched?

Thanks.  As a long time Visual Studio user, I kind of expect CTRL+R to work like CTRL+K where it waits for the next command in the chord.  If CTRL+R is a command alone (especially by default), I honestly don't know how you reconcile that.

What about use Enter as a way to confirm the first shortcut and execute it? That would be different from "Ctrl+R Ctlr+Enter" and it's not possible to have "Ctrl+R+Enter" so there should be no overlapping with other shortcuts

I found the following keybinding can be used to achieve rename behaviour for Ctrl+R Ctrl+R:
{
    "key": "ctrl+r",
    "command": "editor.action.rename",
    "when": "inQuickOpen && inRecentFilesPicker"
}

This overrides the default action for Ctrl+R only when the recent files picker is open (i.e., after the first Ctrl+R keypress) which is workbench.action.quickOpenNavigateNextInRecentFilesPicker, the action that was previously causing VS Code to switch to a different project.

Close as there is a proposed solution above.


Update all files for Ashleyproject submission

Hi Navid and Louis, I would like to submit the final version of all files in my project. (Sorry for the typo in this pull request title, I accidentally hit 'enter' when I tried to type an apostrophe)

On time submission, passing through for evaluation


Issue loading scripts

Expected behavior
Inventories load, steam auth works, and item values are found.
Actual behavior
Inventories are not loading, steam auth returns 404, and item values are not found.
Steps to reproduce the behavior
I changed by websitePort to 81 to solve this error, I have purchased my SteamApi.com key, my bot account has $5 deposited, apache is not installed and as you can see there are no errors in the terminal.
Other

Operating System: Linux (Ubuntu 14)
Node.js version: v8.3.0
How did you start the script: node index.js

Error Stack
Loads of errors. I can see that it's trying to find it through my domain I'm just not sure why or where it's doing that.

Do you access the website with domain:81?

No I just visit domain.com

Well there you have it. Thats the problem. If you arr running with port 81 you should visit - domain.com:81. If port 80 domain.com is fine
Sounds like some kinda web service is using port 80 if you can access your website like that.

What is your URL/domain?

icyskins.com
Adding the ports doesn't appear to work. Could this be a DNS issue?

You have a web server running on port 80.  You have run the script using node index.js then go to icyskins.com:81 and see if it works.
But i will advice you to shut down the web server you are running, and just use port 80 instead.

Any web server should be turned off - unless you know how to add a reverse proxy for this script.
That includes but is not limited to:

Apache
Nginx

I also suggest changing your credentials once you've fixed the web server issue.

Reset SteamApis.com key
Reset Steam API key
Change bot account password and/or reset 2FA


Crash when trying to talk to any NPC

6.0-20221008-104424-6776-1368.zip
When trying to talk with an NPC to say trying to sell stuff u no longer need etc, the game will send out a dialog info crash with the DMP I checked the debugger which read:
GWToolboxdll.dll	GWToolboxdll.dll	C:\Users\dronk\Downloads\GWLauncher\plugins\GWToolboxdll.dll	N/A	Yes	Cannot find or open the PDB file.		36	<IP_ADDRESS>	9/25/2022 5:45 PM	7B330000-7B77F000	6.0-20221008-104424-6776-1368.dmp
I highly doubt I am probably the only one, during Speedruns, etc. I am fine.

Can't copy this issue and crash dump isn't helping much, might  need a video or something

I'll try to make one when I get back for u @3vcloud

https://youtu.be/lN09Ha0MeZs
Video Link for u @3vcloud

Note: I should have added the other crash I had in Marketplace near KC, I can add a video for that too if needed

https://www.youtube.com/watch?v=9TX0w0ax5I4
@3vcloud this one does not have the crash dump on it


feat(imageblock): add props to manage cors and resizing on ImageBlock

General summary
Add props to manage CORS and enable listening to window resizing on ImageBlock component, already on Thumbnail
Screenshots

Check list

[ ] (if has style) Add need: review-integration label
[x] (if has textual documentation) Add need: review-doc label
[x] (if has code) Add need: review-frontend label
[x] (if has react) Check through the react dev check list
[x] (if fix or feature) Add need: test label

Check out the contribution guide for general guidelines for submissions to the design system.

I close this PR to do a new one later to resolve this ticket:
https://lumapps.atlassian.net/browse/LUM-9355


fit_generator used with two generators calling the same keras model

I am using the function fit_generator with different generators for training and testing. These generators, albeit different, are calling the same Keras model.
Here is a minimal example (the code is useless as both train and test generators are the same)
import numpy as np

from keras.layers.core import Masking, Dense, TimeDistributedDense

from keras.layers.recurrent import GRU
from keras.models import Sequential

model = Sequential()
model.add(Masking(mask_value = 0.0, input_shape = (10,2)))
model.add(GRU(2, consume_less = 'gpu', return_sequences = False))
model.add(Dense(2, activation = 'softmax'))
model.summary()

modelBis = Sequential()
modelBis.add(Masking(mask_value = 0.0, input_shape = (10,2)))
modelBis.add(GRU(2, consume_less = 'gpu', return_sequences = True))
modelBis.add(TimeDistributedDense(2))

y = np.asarray([[0,1] for i in range(10)])
def generatorTrain():
    while 1:
        x = np.random.rand(10,10,2)
        prediction = modelBis.predict(x)
        yield prediction, y

def generatorTest():
    while 1:
        x = np.random.rand(10,10,2)
        prediction = modelBis.predict(x)
        yield prediction, y

genTrain = generatorTrain()
genTest = generatorTest()

model.compile('sgd','mse')
model.fit_generator(genTrain, 100, 10, validation_data = genTest, nb_val_samples=100)

After a few epochs, this gives rise to the following error.
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 404, in data_generator_task
    generator_output = next(generator)
  File "<stdin>", line 4, in generatorTest
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 459, in predict
    return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1126, in predict
    batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 846, in _predict_loop
    batch_outs = f(ins_batch)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/theano_backend.py", line 518, in __call__
    return self.function(*inputs)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 871, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 314, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
  File "/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_op.py", line 951, in rval
    r = p(n, [x[0] for x in i], o)
  File "/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_op.py", line 940, in <lambda>
    self, node)
  File "theano/scan_module/scan_perform.pyx", line 76, in theano.scan_module.scan_perform.perform (/home/riminder/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.11-64/scan_perform/mod.cpp:1863)
TypeError: an integer is required
Apply node that caused the error: forall_inplace,gpu,scan_fn}(Shape_i{1}.0, GpuSubtensor{int64:int64:int8}.0, Subtensor{int64:int64:int8}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, gru_2_W, GpuSubtensor{::, int64::}.0, GpuSubtensor{::, :int64:}.0, GpuDimShuffle{x,0}.0)
Toposort index: 42
Inputs types: [TensorType(int64, scalar), CudaNdarrayType(float32, 3D), TensorType(int8, (False, False, True)), CudaNdarrayType(float32, 3D), CudaNdarrayType(float32, (True, False, False)), CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, row)]
Inputs shapes: ['No shapes', 'No shapes', 'No shapes', (10, 10, 2), 'No shapes', (2, 6), 'No shapes', 'No shapes', 'No shapes']
Inputs strides: ['No strides', 'No strides', 'No strides', (20, 2, 1), 'No strides', (6, 1), 'No strides', 'No strides', 'No strides']
Inputs values: [None, None, None, 'not shown', None, 'not shown', None, None, None]
Outputs clients: [[GpuSubtensor{int64:int64:int8}(forall_inplace,gpu,scan_fn}.0, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1})], []]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 656, in fit_generator
    max_q_size=max_q_size)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1412, in fit_generator
    max_q_size=max_q_size)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1474, in evaluate_generator
    'or (x, y). Found: ' + str(generator_output))
Exception: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None

Any idea what is going wrong ?

I would speculate that threading is at fault here, as keras executes generators in their own threads using queues (code). A quick google search reveals that using a theano function, e.g. predict, may not be safely used by multiple threads concurrently (https://github.com/Theano/Theano/issues/2186#issuecomment-63317169).

I agree with @ChristianThomae, you might want your generators to be thread-safe. See this similar issue with elements of solution : #1638

I use keras with tensorflow and got a similar error. My generators for training data and test data end up getting confused when they call the same function (although with different arguments).

not a threading issue, shall focus on the Error happen before that, in this case TypeError: an integer is required shall be root cause.
for my case , it happen to PIL image importError. I have to Import image to solve the same generator output error, which costs me 2 days.
cheers.

Has anyone failed while trying all the solutions described? This proposes  a threadsafe decorator that I tried applying in various forms, even directly to the predict function of the model. A solution on StackOverflow also proposes pre-calling model._make_predict_function() which does not work for me either. I only use one worker for fit_generator and have played with pickle_safe as well. I have described our specific issue here:
https://github.com/costapt/vess2ret/issues/5


Missing Details on User Experience

OneNote is one of the most challenging aspects of migrating to OneDrive. This page could use some details on what the experience for the user is going to be and the additional actions that they need to take.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9fff09a1-5bc3-a9ea-ace0-5f3e865efbe2
Version Independent ID: 08a51327-644d-9099-efb2-b2d7523b2e1b
Content: Migrating OneNote folders with the SharePoint Migration Tool (SPMT) - Migrate to Microsoft 365
Content Source: migration/migrate-onenote-spmt.md
Service: sharepoint-online
Sub-service: sharepoint-migration
GitHub Login: @JoanneHendrickson
Microsoft Alias: jhendr

Yes, users need to disconnect their notebook(s) from the OneNote desktop app and then reconnect from its new home in OneDrive.

Yes, users need to disconnect their notebook(s) from the OneNote desktop app and then reconnect from its new home in OneDrive.

Thanks for clarifying, I'll propose adding a reference to that in this article, if it gets approval, it will be in the published version of the article.


Tf-slim: Unable to read dataset using slim.data_set_provider.DataSetProvider when images are not in JPEG.

What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?
Couldn't find relevant threads as there aren't many tf-slim questions.
Environment info
Operating System:
Ubuntu 16.04
Installed version of CUDA and cuDNN:
8.00
If installed from binary pip package, provide:

A link to the pip package you installed:
The output from python -c "import tensorflow; print(tensorflow.__version__)".
0.11

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
Basically I have directory of subdirectories, with each subdirectory containing png images of a certain class. Editing the tf-slim download_and_convert_flowers.py to suit my images, I created a set of tfrecord files (train and validation both included) and stored it in a directory.
Following which, I used the 'get_split' function from dataset_utils in https://github.com/tensorflow/models/blob/master/slim/datasets/dataset_utils.py
to create a DataSet class from reading the tfrecord files of a certain type (either train or validation. I indicated 'train'). So now I have a DataSet class to read.
The problem comes when I try to use a batch loading function to actually start reading the DataSet object for extracting images and creating a batch:
def load_batch(dataset, batch_size=32, height=299, width=299, is_training=False):
    """Loads a single batch of data.

    Args:
      dataset: The dataset to load.
      batch_size: The number of images in the batch.
      height: The size of each image after preprocessing.
      width: The size of each image after preprocessing.
      is_training: Whether or not we're currently training or evaluating.

    Returns:
      images: A Tensor of size [batch_size, height, width, 3], image samples that have been preprocessed.
      images_raw: A Tensor of size [batch_size, height, width, 3], image samples that can be used for visualization.
      labels: A Tensor of size [batch_size], whose values range between 0 and dataset.num_classes.
    """
    data_provider = slim.dataset_data_provider.DatasetDataProvider(
        dataset)
        # common_queue_capacity=32)
        # common_queue_min=8)
    image_raw, label = data_provider.get(['image', 'label'])

    # Preprocess image for usage by Inception.
    image = inception_preprocessing.preprocess_image(image_raw, height, width, is_training=is_training)

    # Preprocess the image for display purposes.
    image_raw = tf.expand_dims(image_raw, 0)
    image_raw = tf.image.resize_images(image_raw, [height, width])
    image_raw = tf.squeeze(image_raw)

    # Batch it up.
    images, images_raw, labels = tf.train.batch(
          [image, image_raw, label],
          batch_size=batch_size,
          num_threads=1,
          capacity=2 * batch_size)

    return images, images_raw, labels

The problem comes from slim.dataset_data_provider.DatasetDataProvider as I couldn't read the dataset at all.
Here is my error traceback:
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
name: GeForce GTX 860M
major: 5 minor: 0 memoryClockRate (GHz) 1.0195
pciBusID 0000:01:00.0
Total memory: 3.95GiB
Free memory: 3.60GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 860M, pci bus id: 0000:01:00.0)
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 1898 get requests, put_count=1100 evicted_count=1000 eviction_rate=0.909091 and unsatisfied allocation rate=1
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 100 to 110
INFO:tensorflow:Starting Session.
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
Not a JPEG file: starts with 0x89 0x50
Not a JPEG file: starts with 0x89 0x50
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors.InvalidArgumentError'>, Invalid JPEG data, size 26498
	 [[Node: case/If_0/DecodeJpeg = DecodeJpeg[acceptable_fraction=1, channels=3, fancy_upscaling=true, ratio=1, try_recover_truncated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](case/If_0/DecodeJpeg/Switch:1, ^case/Assert/AssertGuard/Merge/_7825)]]

Caused by op u'case/If_0/DecodeJpeg', defined at:
  File "code.py", line 148, in <module>
    images, images_raw, labels = load_batch(dataset, height = image_size, batch_size = batch_size, width = image_size, is_training = True)
  File "code.py", line 27, in load_batch
    dataset)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/dataset_data_provider.py", line 78, in __init__
    tensors = dataset.decoder.decode(data, items)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 398, in decode
    outputs.append(handler.tensors_to_item(keys_to_tensors))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 297, in tensors_to_item
    image = self._decode(image_buffer, image_format)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 324, in _decode
    }, default=decode_jpg, exclusive=True)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2886, in case
    case_seq = _build_case()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2868, in _build_case
    name="If_%d" % i)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 1710, in cond
    orig_res, res_t = context_t.BuildCondBranch(fn1)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 1613, in BuildCondBranch
    r = fn()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 317, in decode_jpg
    return image_ops.decode_jpeg(image_buffer, self._channels)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_image_ops.py", line 283, in decode_jpeg
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Invalid JPEG data, size 26498
	 [[Node: case/If_0/DecodeJpeg = DecodeJpeg[acceptable_fraction=1, channels=3, fancy_upscaling=true, ratio=1, try_recover_truncated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](case/If_0/DecodeJpeg/Switch:1, ^case/Assert/AssertGuard/Merge/_7825)]]

W tensorflow/core/framework/op_kernel.cc:968] Out of range: FIFOQueue '_0_batch/fifo_queue' is closed and has insufficient elements (requested 3, current size 0)
	 [[Node: batch = QueueDequeueMany[_class=["loc:@batch/fifo_queue"], component_types=[DT_FLOAT, DT_FLOAT, DT_INT64], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, batch/n)]]
W tensorflow/core/framework/op_kernel.cc:968] Out of range: FIFOQueue '_0_batch/fifo_queue' is closed and has insufficient elements (requested 3, current size 0)
	 [[Node: batch = QueueDequeueMany[_class=["loc:@batch/fifo_queue"], component_types=[DT_FLOAT, DT_FLOAT, DT_INT64], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, batch/n)]]
Traceback (most recent call last):
  File "code.py", line 171, in <module>
    number_of_steps = 10)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 780, in train
    raise
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 969, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 797, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 386, in join
    six.reraise(*self._exc_info_to_raise)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/queue_runner.py", line 225, in _run
    sess.run(enqueue_op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 915, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 985, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Invalid JPEG data, size 26498
	 [[Node: case/If_0/DecodeJpeg = DecodeJpeg[acceptable_fraction=1, channels=3, fancy_upscaling=true, ratio=1, try_recover_truncated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](case/If_0/DecodeJpeg/Switch:1, ^case/Assert/AssertGuard/Merge/_7825)]]

Caused by op u'case/If_0/DecodeJpeg', defined at:
  File "code.py", line 148, in <module>
    images, images_raw, labels = load_batch(dataset, height = image_size, batch_size = batch_size, width = image_size, is_training = True)
  File "code.py", line 27, in load_batch
    dataset)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/dataset_data_provider.py", line 78, in __init__
    tensors = dataset.decoder.decode(data, items)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 398, in decode
    outputs.append(handler.tensors_to_item(keys_to_tensors))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 297, in tensors_to_item
    image = self._decode(image_buffer, image_format)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 324, in _decode
    }, default=decode_jpg, exclusive=True)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2886, in case
    case_seq = _build_case()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2868, in _build_case
    name="If_%d" % i)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 1710, in cond
    orig_res, res_t = context_t.BuildCondBranch(fn1)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 1613, in BuildCondBranch
    r = fn()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 317, in decode_jpg
    return image_ops.decode_jpeg(image_buffer, self._channels)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_image_ops.py", line 283, in decode_jpeg
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Invalid JPEG data, size 26498
	 [[Node: case/If_0/DecodeJpeg = DecodeJpeg[acceptable_fraction=1, channels=3, fancy_upscaling=true, ratio=1, try_recover_truncated=false, _device="/job:localhost/replica:0/task:0/cpu:0"](case/If_0/DecodeJpeg/Switch:1, ^case/Assert/AssertGuard/Merge/_7825)]]

Upon inspection, I have checked that a likely error comes from the line   File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 297, in tensors_to_item     image = self._decode(image_buffer, image_format)
in tfexample_decoder.py, the image_format, if not indicated, will be a JPEG image by default unless there are 4 channels (RGBA) in the images. But since my images are grayscale, a JPEG decoder is used by default. Yet, I have no way to specify the image format as 'png' specifically unless the source code is changed. How should I go about this, or am I mistaken about the error I have arrived at?
I have previously successfully run the code (many of which are referenced from the tf-slim walkthrough ipynb file), but the images I used were in jpeg.
Any help is very much appreciated. Thank you for your time.

@kwotsin would you be able to do what you want to do just by using the core (i.e. non-contrib) APIs?
@sguada This issue makes a compelling case that there's an API design issue in contrib/slim that prevents the user from loading grayscale PNG files. It's a niche use case, but looks like something that ought to be fixed, if you have time to help our friend.

tf-slim.data can decode jpeg, png or raw see here
The only caveat is that it needs to know that the encoded image has png format when creating the tf-example.
When you created the dataset did you specify the image was a png?  So for instance the flowers dataset assumes that the images are jpg

@sguada Yes I have changed the code to read the images in png format before I read them using the slim API, however I can't seem to find a way to get tf-slim to know the files are encoded in png.
@jart I am figuring out how to do this in pure TF, but it seems I might have to rewrite a custom function for reading the dataset or simply read from disk the images (might be much slower). But for the time being I have bypassed the problem by trying out jpeg files first.

Just change 'jpg' to 'png' in this line when the file is 'png'
https://github.com/tensorflow/models/blob/master/slim/datasets/download_and_convert_flowers.py#L146
That way it would store the image format and it would be able to decode it.

@kwotsin , based on @sguada 's reply above, it seems that tf-slim does support png as you want and the problem was with the sample download_and_convert_flowers.py script.
Closing this out, though feel free to reopen if I misunderstood.


Bump to latest versions

Would have been really awesome to have a Heroku review app deployed for this PR, and see if it works alright before merging for the main app! :)

Yep - will work well with the tugboat stuff we're also working on in Backstage too.


Results after training for 300 epochs not satisfying

After training for 300 epochs with the standard configuration on the cityscapes dataset, except for the crop size, which I had to downsize to 256x256, the result is just a noise of some random pixel values. Does anyone encounter the same issues?

Do you know how to solve this problem?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "F:/SemicycleGAN0/Semi-supervised-segmentation-cycleGAN-master/main.py", line 87, in
main()
File "F:/SemicycleGAN0/Semi-supervised-segmentation-cycleGAN-master/main.py", line 73, in main
model.train(args)
File "F:\SemicycleGAN0\Semi-supervised-segmentation-cycleGAN-master\model.py", line 439, in train
fake_gt_discriminator = make_one_hot(fake_gt_discriminator, args.dataset, args.gpu_ids)
File "F:\SemicycleGAN0\Semi-supervised-segmentation-cycleGAN-master\utils.py", line 347, in make_one_hot
one_hot = torch.FloatTensor(labels.size(0), C, labels.size(2), labels.size(3)).zero_()
IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)

@mojoee Hi, I am training it on Cityscapes. After around 200 epoches, mIoU is only around 20%. Did you figure out how to address this issue? Many thanks.


fix urls to image and blog-post of 'hosting owncast behind nginx'

the links to my blog changed, due to a dumb refactoring of content and resources I made. This broke the links and caused them to 404. This PR replaces the 404'ing links with actually working links (same content behind the article as before)

haha we've all been there. Thanks for the update!


[EngSys] Bump extensions package references

Sumary
The focus of these changes is to move the extensions package versions to the latest and bump associated BCL extensions due to security scanning flags.

API change check
API changes are not detected in this pull request.

//cc: @m-redding for awareness.    We're probably going to hit some packages that you've got planned.

/azp run net - eventhub - functions - ci

/azp run net - eventhub - functions - tests


ocaml-ng.ocamlPackages_4_0[2-5].ocaml: unbreak on aarch64-linux

Description of changes
Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@ofborg build ocaml-ng.ocamlPackages_4_02.ocaml ocaml-ng.ocamlPackages_4_03.ocaml ocaml-ng.ocamlPackages_4_04.ocaml ocaml-ng.ocamlPackages_4_05.ocaml

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/1721


fix doc regression [ci-skip]

Overview
Fixes regression in CONTRIBUTING.rst
Related Issue / Discussion
None
Additional Information
None
Contribution Agreement
(as per https://github.com/geopython/pywps/blob/master/CONTRIBUTING.rst#contributions-and-licensing)

[x] I'd like to contribute [feature X|bugfix Y|docs|something else] to PyWPS. I confirm that my contributions to PyWPS will be compatible with the PyWPS license guidelines at the time of contribution.
[x] I have already previously agreed to the PyWPS Contributions and Licensing Guidelines

Coverage decreased (-0.1%) to 74.881% when pulling 3f53f89f9455a14c0965321ce61eca1b7ef0d614 on tomkralidis:fix-contributing into e10f641e2550577636baff67e6bd474a135e38a2 on geopython:master.

Coverage decreased (-0.1%) to 74.881% when pulling 3f53f89f9455a14c0965321ce61eca1b7ef0d614 on tomkralidis:fix-contributing into e10f641e2550577636baff67e6bd474a135e38a2 on geopython:master.


Generate OSCORE object and use it for endpoint creation

Code to implement the points listed here:
https://github.com/eclipse/leshan/pull/718#issuecomment-521316754

create the OSCORE object
use it in leshan client demo (keeping hardcoded value but create oscore object)
modify ServersInfoExtractor and ServerInfo to support oscore.
use ServerInfo in CaliforniumEndpointsManager to create endpoint with oscore support.

The OSCORE object is generated and filled in the Leshan client demo. It will then be used to create a ServerInfo object using the ServersInfoExtractor. Next the ServerInfo is used in the CaliforniumEndpointsManager to generate endpoints towards servers that OSCORE should be used with.
See also issue regarding OSCORE Leshan client integration:
https://github.com/eclipse/leshan/issues/726

I suppose we should add a new attribute for OSCORE which could be used in addition of existing DTLS identity (in case of OSCORE over coaps) or without it (in case of OSCORE over coap).

Yes, that makes sense.

In oscore what could be the value which could be used as an identifier ? I need to read the rfc again (intuitively I suppose that Sender ID or/and Recipient ID could be candidate?)

A combination of the sender ID, recipient ID and ID context should identify a pair of endpoints communicating using OSCORE.

You should have a look at §5.5.5 - Endpoint Client Name about you "identity" concern.
About that is sender ID / receiver ID available in californium endpoint context ?

About that is sender ID / receiver ID available in californium endpoint context ?

Yes it is available but only from Californium 2.0.0-M17. Since the context DB is a singleton in the Californium we are using now the values could also be gotten from there (for now).

You should have a look at §5.5.5 - Endpoint Client Name about you "identity" concern.

I now updated the code so that the Sender ID of the client is equal to its name (endpoint). So if I start a client with name "c1" its sender ID will be 0x6331, and on the server side in the web interface the sender ID must be entered as "6331" (hexadecimal). Done in commit 401db2bfd59b9481cc5f452a524b7c52bdb84f19

For further work on the identity I would perhaps wait until after updating to Californium 2.0.0-M17. There the sender ID, recipient ID and ID context are conveniently available in the endpoint context.

I had to do one more commit regarding the registration, Identity and SecurityChecker. Previously I could not register an OSCORE client using the name I configured it with in the web interface of the server. Since it counted as not secure it but yet had a non-null securityInfo it wanted me to use DTLS. I fixed this in commit 40213f6bfc1fae4dd4c0d51eb3d48d17a1cfbd5a
The SecurityChecker should be expanded with checks to the identity when using OSCORE. Also the Identity class expanded to support OSCORE-related identities. SecurityInfo already has a useOSCore flag.

About CaliforniumEndpointsManager, maybe I missed something but the idea is to begin by supporting Oscore without DTLS right ?
So I think the oscore stuff is not at "The Good Place".

For further work on the identity I would perhaps wait until after updating to Californium 2.0.0-M17. There the sender ID, recipient ID and ID context are conveniently available in the endpoint context.

makes sense !

About CaliforniumEndpointsManager, maybe I missed something but the idea is to begin by supporting Oscore without DTLS right ?

Yes, my intention was to start by supporting OSCORE without DTLS. So the good place would be outside of the if(serverInfo.isSecure()), like down by row 220?

So the good place would be outside of the if(serverInfo.isSecure()), like down by row 220?

yep :)

yep :)

Okay, sounds good :)
I have now done this in commit 7116077e8e6ed4a1a8456b9040333ad01875b146
Perhaps at some later stage it could be extracted to a separate method since we may want to combine the OSCORE configuration of an endpoint with different DTLS modes in a convenient way and minimize code reuse.

Perhaps at some later stage it could be extracted to a separate method since we may want to combine the OSCORE configuration of an endpoint with different DTLS modes in a convenient way and minimize code reuse.

Correct, we will see when we will add this.
I think we are good, do you plan to add more stuff in this PR ? (maybe just fix formatting issue?)

I think we are good, do you plan to add more stuff in this PR ? (maybe just fix formatting issue?)

Yes, fixing the formatting was the last thing I had planned. I will proceed with that.

I think we are good, do you plan to add more stuff in this PR ? (maybe just fix formatting issue?)

I have now applied the Leshan Eclipse formatter to the parts of the code I have changed. Done in commit 0138768e1b906ad8f81f67389103266832360d66

I squashed all your commits in one (e31012710a149b619852f6b741e8ea4da083f3e6) which is now integrated in oscore branch.
I tried a "mvn clean install" and that failed so I fixed issues, see commit cc6136a63f081cfa8f8280a6ef67ff53a84288ee
Do not hesitate to launch this often, tests is here to help you ;)
About formatting I find several more issues, see commit e2ae4bca7cffca3430876a732a793defebf1b22d.
Are you using eclipse ? Did you follow this guide. In theory "format on save" is activated per project, so it should be very difficult to write bad formatted code :thinking:.
What do you plan as next steps ? :)

I squashed all your commits in one (e310127) which is now integrated in oscore branch.

Thanks!

I tried a "mvn clean install" and that failed so I fixed issues, see commit cc6136a
Do not hesitate to launch this often, tests is here to help you ;)

Yes, I was using Eclipse and not running the maven tests from the command line like that. I should have ran tests more frequently and will try to do so in the future.

About formatting I find several more issues, see commit e2ae4bc.
Are you using eclipse ? Did you follow this guide. In theory "format on save" is activated per project, so it should be very difficult to write bad formatted code .

Yes, I am using Eclipse. Unfortunately up until now I have not been using the "format on save" feature. But rather formatting manually with CTRL+SHIFT+F. I will start using "format on save" as you suggest.

What do you plan as next steps ? :)

Some thoughts for next steps:

Update code to Californium 2.0.0-M17
Implement an Identity and related things in SecurityChecker for OSCORE
Work on the server side issues according to discussion in https://github.com/eclipse/leshan/issues/727
Add possibility of entering OSCORE context information as command line parameters for client

What do you think makes most sense to prioritize?

This order makes sense to me :)

This order makes sense to me :)

Okay, sounds good. I will proceed accordingly. :)


params and states should be accessed via a "params" and "states" method

Currently if you declare a param or state it creates a instance method of the same name.
This can cause name conflicts with other methods, and also does not tell readers of the code what the special properities of the methods are.
So the suggestion is that we access states and properties via a method called states / params.  For example
defined_state :foo
required_param :bar
...
    state.foo # currently can just say foo
    param.bar # currently can just say bar
    # alternative syntax 1
    states.foo
    params.bar
    # alternative syntax 2
    state_foo
    param_bar
...

Questions for discussion:

should the method be singular or plural, or just create the method like state_foo, param_bar
should we deprecate the current capability?  Why / why not?

I like state.foo / param.bar. I'm not sure about singular or plural. I'm experiencing some cognitive dissonance here. I think singular state and plural params reads most natural, but having the inconsistency is clearly wrong.

Yes. We don't have to remove it right away, but I don't think we need to support both interfaces for the long term.

I think the problem is you talk about the state (singular) and the parameters (plural) of a component.  State is kind of like sheep or deer.  I could go with state/params

@catmando So sheeps.foo / deers.bar? Okay okay, enough tom foolery. That's a good point though. I'd be okay with state/params. The next question is should params be props?

yeah I don't know... I personally much prefer params (they just feel more like parameters than properties) but not sure how much confusion it makes.

params gives a nod to rails, and will be familar to people coming from that background
params is what it is today and works
params allows "props" to be for low level access to the actual react props hash (ugh)

params makes it harder for react.js people (but not a lot I think)
params is longer than props

I agree. Params are beefed up props. The fact that they have more behavior than the underlying react props justifies making a distinction. Plus, it allows direct access to low level props if you need.

I agree @ajjahn, they have different behavior so should be kept distinct; I also prefer it with the the . as it feels more natural in my head and less 'magical'.

FYI... this also solves a nagging problem that we can't easily call a param "params" which react-router wants to do.  Now that params are scoped by the params method, and params.params works fine.

state/params looks good to me.  Would still keep [] access too?

I don;t think so

This issue was moved to reactrb/reactrb#83


Releases and branches

We like the idea of the plugin, but it seems like every release (to nuget) it's hard to find the corresponding source code. Is it possible to tag/release all released versions ?
Thanks in advance,
Bart,

Thanks for the suggestion. I will look into some automation perhaps.


AutoPr-SecurityCenter-chlahav-REST Spec PrNumber 5435

Generated from https://github.com/azure/azure-rest-api-specs/pull/5435

@azuresdkci Retest this please


Error trying to compile

Hey! I'm trying to compile your code on Ubuntu 17.10 and I get the following error when trying to run compile.sh
clang: error: no such file or directory: 'external/lib/zany/*.c
I'm assuming there's meant to be another directory?

Sorry, should be fixed now!

Can confirm -- compiles for me now.

Thank you! Does it run as well? (With the example .prg?)
./enias examples/hackman/hackman.prg

It works -- but I am not sure what the output is meant to be.
On Mon, Feb 26, 2018 at 12:19 AM Peter Bjorklund<EMAIL_ADDRESS>wrote:

Thank you! Does it run as well? (With the example .prg?)
./enias examples/hackman/hackman.prg
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/Piot/enias/issues/1#issuecomment-368401797, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACSSX-OfP9k-dG6746X4yY6ChOd5SbR0ks5tYkzcgaJpZM4SSNaW
.

No, it is very early days for this project. The hackman.prg is mainly just testing the sprites and tile background for now. 😊
Next thing is to include the sound chip.
Thanks for helping out, I appreciate it!

No problem! I want to see more fantasy consoles get off the ground!


Security object not lwm2m specification compliant

Hello,
to me it seems that the Security Object as implemented in Mbed-Cloud-Client is not compliant to the Lwm2m Specification. Mbed-Cloud-Client defines the following resources for the Security Object: https://github.com/PelionIoT/mbed-cloud-client/blob/master/mbed-client/mbed-client/m2msecurity.h#L38-L58
The Lwm2m Object Registry on the other hand uses those: http://www.openmobilealliance.org/tech/profiles/LWM2M_Security-v1_0.xml
Consider the difference for resource with id 12 (Bootstrap-Server Account Timeout vs OpenCertificateChain).
Best regards,
Sebastian

The additionally added resources (OpenCertificateChain, CloseCertificateChain, ReadDeviceCertificateChain) conflict with resources that were introduced in Lwm2m 1.1 (http://openmobilealliance.org/release/ObjLwM2M_Security/V1_1-20180612-C/OMA-SUP-XML_LWM2M_Security-V1_1-20180612-C.xml)

Hi @ghseb thanks for the report. We can review this implementation as part of future work towards new specifications.

Hi @ghseb thanks for the report. We can review this implementation as part of future work towards new specifications.

Thank you, I think this would simplify usage of cloud-client a lot (especially when it comes to bootstrapping).
Do you mind if this issue is left open as a reminder? If you aware of it I also could close it (I assume an eventual change would be rather long term).

Thank you, I think this would simplify usage of cloud-client a lot (especially when it comes to bootstrapping).
Do you mind if this issue is left open as a reminder? If you aware of it I also could close it (I assume an eventual change would be rather long term).

You can leave it open as a remainder.

You can leave it open as a remainder.

PDMC 4.12.0 contains a fix for this.
In this version, we introduced LwM2M compliance feature.
The feature is enabled by LWM2M_COMPLIANT compilation flagץ
When the feature is enabled, the OpenCertificateChain, CloseCertificateChain, ReadDeviceCertificateChain aren't in use.


Gizmo not respond to the rotation or translation

When manipulating the gizmo, the gizmo doesn't move with the drag.

Can you provide any more details? Does this happen with the provided minimal example?

Here you can see the gizmo didn't rotate with the drag.
https://user-images.githubusercontent.com/771776/150466907-499f7985-8968-4957-8b6e-384d0cbfd370.mp4


pandas html

Implement https://pandas.pydata.org/docs/reference/io.html#html as a data saver/loader.

It should go into pandas_extensions.py, one class for the writer, on class for the reader, e.g. much like how the pandas pickle reader and writer are structured.
There should be requisite tests to exercise the functionality. Note: new style. We want a single test that exercises the save and then load functionality, using what was just saved.
Then an example should be added to the example/pandas/materialization example. It should just add to the existing materializers; yes this only exercises the writer (i.e. saver).
If there's an issue with type hints, let me know and we can chat through it.

I'll be working on this ticket.

Implemented!


Align "No node pools" text colour

The "No node pools" message is currently displayed in yellow, the colour we reserve for warnings. We should align it to use the same colour as the "No apps" and "No client certificates" messages.

:+1:


Add Menoh to the list of applications using MKL-DNN

Preferred Networks recently released Menoh, an inference engine for running ONNX model using MKL-DNN.
Add a link to Menoh.

Thanks!


[Misc]: Assertion with no scription in vllm with DeepSeekMath 7b model, why, how to fix?

Anything you want to discuss about vllm.
I'm working with the DeepSeekMath 7b model to generate synthetic data using Python, but I'm encountering an AssertionError with no description alongside warnings related to token length exceeding the model's limit. The relevant portion of my code triggers this issue during a batch operation with llm.generate(...). Here’s the code snippet and the error logs:
request_outputs: list[RequestOutput] = gen.llm.generate(batch_prompts, gen.sampling_params)

This is part of a larger function that processes datasets, initiated by:
lst: list[dict] = gen_data_set_from_ans_str_2_jsonl(gen, path_2_src_dataset, output_path, prompt_template, num_data_gens_per_txt_files=num_data_gens_per_txt_files)

Here are the logs indicating the error and warnings:
AssertionError:
len(all_raw_data)=274
len(all_raw_data)/batch_size = 1/9223372036854775807 = 1
num_batches=1
Processed prompts:   1%|██▌ | 2/274 [00:10<19:40,  4.34s/it]
WARNING 05-15 19:44:53 scheduler.py:619] Input prompt (4557 tokens) is too long and exceeds limit of 4096
Processed prompts:  24%|███████████ | 66/274 [01:41<03:38,  1.05s/it]

The process always halts at 66 out of 274. I suspect the issue relates to handling input sizes, but I’m unclear why the assertion fails without any description or how to handle these warnings properly. Has anyone faced similar issues, or can anyone suggest how to debug or fix this situation?

Code:
import math
import os
from pathlib import Path
from tqdm import tqdm

import torch

import json
from typing import Optional
import sys
from pprint import pprint
from vllm import LLM, SamplingParams, CompletionOutput, RequestOutput

import tenacity

# -- Mathematica gen prompt

# HELM_MATH_PROMPT_8SHOT_COT2_GEN_SYN_METHEMATICA_TEMPLATE: str = (
# """Given a mathematics problem, its answer, math category, and problem type, generate a solution to the problem, with step by step reasoning that justifies the answer.
# Simplify your answer as much as possible. Always give the final answer inside a \\boxed{answer}.###
# This is the format:
# Problem: Let $r=3^s-s$ and $s=2^n+1$. What is the value of $r$ when $n=2$?
# Solution: Let's think step by step. First substitute $n=2$ into the expression for $s$ to find $s=2^2+1=5$. Then substitute $s=5$ into the expression for $r$ to find $r=3^5-5=243-5=\\boxed{238}. The final answer is: \\boxed{238}.###
# Problem: If $x^{2y}= 4$ and $x = 4$, what is the value of $y$? Express your answer as a common fraction.
# Solution: Let's think step by step. Plugging $x = 4$ into the first equation, we get $4^{2y} = 4^1 \\Rightarrow 2y = 1 \\Rightarrow y = \\boxed{\\frac{1}{2}}. The final answer is: \\boxed{\\frac{1}{2}}.###
# Problem: If $y = \\displaystyle\\frac{1}{3x+1}$, what is the value of $x$ when $y = 1$?
# Solution: Let's think step by step.Since $y=1$, we have $1 =\\displaystyle\\frac{1}{3x+1}$. Multiplying both sides by $3x+1$, we have $$3x+1=1$$ $$\\Rightarrow \\qquad 3x=0$$ $$\\Rightarrow \\qquad x=\\boxed{0}$$. The final answer is: \\boxed{0}.###
# Problem: A scale drawing of a park shows that one inch represents 800 feet. A line segment in the drawing that is 4.75 inches long represents how many feet?
# Solution: Let's think step by step. Each inch of the 4.75-inch line segment represents 800 feet, so the whole line segment represents $4.75\\times800=\\frac{19}{4}\\cdot800=19\\cdot200=\\boxed{3800}$ feet. The final answer is: \\boxed{3800}###
# Problem: If $(x + y)^2 = 25$ and $xy = 6$, what is the value of $x^2 + y^2$?
# Solution: Let's think step by step. We know that $(x + y)^2 = (x^2 + y^2) + 2xy = 25$. We are given that $xy = 6$. So, by substitution, $x^2 + y^2 + 2xy = x^2 + y^2 + 2(6) = 25$. It follows that $x^2 + y^2 = 25 - 12 = \\boxed{13}$.  Assuming she keeps up that rate of consumption, how many Popsicles can Megan finish in 4 hours and 30 minutes?
# Solution: Let's think step by step. Let $p$ be the number of Popsicles Megan can finish in 4 hours and 30 minutes. If we convert that period of time into minutes, we find that 4 hours and 30 minutes is equal to $(4)(60)+30=270$ minutes. From here, we can set up the proportion \\begin{align*} \\frac{x}{270}& =\\frac{1}{15}\\\\\\Rightarrow \\qquad x& =\\left(\\frac{1}{15}\\right)(270)\\\\\\Rightarrow \\qquad x& =\\boxed{18}\\end{align*}. The final answer is: \\boxed{18}###
# Problem: Compute $95^2$ in your head.
# Solution: Let's think step by step. We have $(90 + 5)^2 = 90^2 + 2(90)(5) + 5^2 = 8100 + 900 + 25 = \\boxed{9025}$. The final answer is: \\boxed{9025}.###
# Problem: If $2^8=16^x$, find $x$.
# Solution: Let's think step by step. We can write $16$ as $2^4$. Therefore, we can write our equation as $2^8 = 2^{4 \\cdot x}$. Solving, we get that $x = \\boxed{2}$. The final answer is: \\boxed{2}.###
# This is the new problem to generate a solution string from the answer, problem, category, and problem type:
# Problem: {problem}
# The answer is {answer}, the math category is {category}, the problem type is {prob_type}.
# Solution: Let's think step by step.""")

# HELM_MATH_PROMPT_8SHOT_COT2_USE_SOLN_CODE_GEN_SYN_METHEMATICA_TEMPLATE: str = (
# """Given a mathematics problem, its answer, math category, problem type, and mathematica code that describes the solution to the answer, generate a solution to the problem, with step by step reasoning followed by the answer.
# Simplify your answer as much as possible. Always give the final answer inside a \\boxed{answer}.###
# This is the format:
# Problem: Let $r=3^s-s$ and $s=2^n+1$. What is the value of $r$ when $n=2$?
# Solution: Let's think step by step. First substitute $n=2$ into the expression for $s$ to find $s=2^2+1=5$. Then substitute $s=5$ into the expression for $r$ to find $r=3^5-5=243-5=\\boxed{238}. The final answer is: \\boxed{238}.###
# Problem: If $x^{2y}= 4$ and $x = 4$, what is the value of $y$? Express your answer as a common fraction.
# Solution: Let's think step by step. Plugging $x = 4$ into the first equation, we get $4^{2y} = 4^1 \\Rightarrow 2y = 1 \\Rightarrow y = \\boxed{\\frac{1}{2}}. The final answer is: \\boxed{\\frac{1}{2}}.###
# Problem: If $y = \\displaystyle\\frac{1}{3x+1}$, what is the value of $x$ when $y = 1$?
# Solution: Let's think step by step.Since $y=1$, we have $1 =\\displaystyle\\frac{1}{3x+1}$. Multiplying both sides by $3x+1$, we have $$3x+1=1$$ $$\\Rightarrow \\qquad 3x=0$$ $$\\Rightarrow \\qquad x=\\boxed{0}$$. The final answer is: \\boxed{0}.###
# This is the new problem to generate a solution string from the answer, problem, category, and problem type:
# Problem: {problem}
# The answer is {answer}, the math category is {category}, the problem type is {prob_type}, and the mathematica code is {mathematica_code}.
# Generate a solution using that information:
# Solution: Let's think step by step.""")

# # Goal of prompt is to generate (missing) solution strings from a the mathematica code, given 1 few shot example. ref: https://chatgpt.com/c/fb17c070-bd1e-4aad-9078-88c44abc4ae9
# mathematica_code1: str = open(os.path.expanduser('~/gold-ai-olympiad/py_src/training/algebraic_manipulation.nb')).read()
# MATHEMATICA_SYNTH_GEN_PROMPT_TEMPLATE: str = (
# f"""
# Given a mathematics problem, its answer, math category, problem type, and mathematica code that describes the solution to the answer, generate a solution to the problem, with step by step reasoning followed by the answer.
# Simplify your answer as much as possible. Always give the final answer inside a \\boxed{{answer}}.###
# Problem: Let $r=3^s-s$ and $s=2^n+1$. What is the value of $r$ when $n=2$?
# Answer: 238
# Math Category: Algebra
# Problem Type: Algebraic Manipulation
# Mathematica Code: {mathematica_code1}
# Solution: Let's think step by step. First substitute $n=2$ into the expression for $s$ to find $s=2^2+1=5$. Then substitute $s=5$ into the expression for $r$ to find $r=3^5-5=243-5=\\boxed{238}. The final answer is: \\boxed{238}.###
# Problem: {{problem}}
# Answer: {{answer}}
# Math Category: {{category}}
# Problem Type: {{prob_type}}
# Mathematica Code: {{mathematica_code}}
# Solution: Let's think step by step."""
# )
# Goal of prompt is to generate (missing) solution strings from a the mathematica code, given 1 few shot example. ref: https://chatgpt.com/c/fb17c070-bd1e-4aad-9078-88c44abc4ae9
MATHEMATICA_SYNTH_GEN_PROMPT_TEMPLATE: str = (
f"""
Given a mathematics problem, its answer, math category, and problem type that describes the solution to the answer, generate a solution to the problem, with step by step reasoning followed by the answer.
Simplify your answer as much as possible. Always give the final answer inside a \\boxed{{answer}}.###
Problem: Let $r=3^s-s$ and $s=2^n+1$. What is the value of $r$ when $n=2$?
Answer: 238
Math Category: Algebra
Problem Type: Algebraic Manipulation
Solution: Let's think step by step. First substitute $n=2$ into the expression for $s$ to find $s=2^2+1=5$. Then substitute $s=5$ into the expression for $r$ to find $r=3^5-5=243-5=\\boxed{238}. The final answer is: \\boxed{238}.###
Problem: {{problem}}
Answer: {{answer}}
Math Category: {{category}}
Problem Type: {{prob_type}}
Solution: Let's think step by step."""
)

# --

@tenacity.retry(stop=tenacity.stop_after_attempt(10), wait=tenacity.wait_exponential(multiplier=2, max=64))
def gen_synthetic_solution(gen, prompt_template: str, problem, answer, category, prob_type = None) -> str:
    """
    Generates a synthetic solution to a math problem using a language model.

    Args:
        gen (GPT): The language model used to generate the solution.
        problem (str): The math problem.
        answer (str): The correct answer to the problem.
        category (str): The category of the problem.
        prob_type (str): The type of problem.

    Returns:
        str: The synthetic solution to the problem.
    """
    prompt: str = prompt_template.replace('{problem}', problem)
    # solution = generate_completion(problem, answer, category, prob_type)
    response: dict = gen.llm.chat.completions.create(
        model=gen.model,
        messages=[
            {"role": "system", "content": gen.system_prompt},
            {"role": "user", "content": prompt},
        ],
        temperature=gen.sampling_params.temperature,
        top_p=gen.sampling_params.top_p,
        n=gen.sampling_params.n,
        stop=gen.sampling_params.stop[:3],
        )
    solution: str = response.choices[0].message.content
    return solution

def extract_problem_data(file_path: str) -> dict[str, str]:
    """
    Extracts the problem and answer from a .txt file.

    Args:
        file_path (str): The path to the .txt file containing the problem.

    Returns:
        Dict[str, str]: A dictionary with keys 'problem' and 'answer'.
    """
    with open(file_path, 'r', encoding='utf-8') as file:
        content = file.read().split("\nAnswer:\n")
        problem = content[0].strip()
        answer = content[1].strip()
    return {'problem': problem, 'solution': '', 'answer': answer}

def gen_data_set_from_ans_str_2_jsonl(
                                        gen,
                                        source_path: str,
                                        output_path: str,
                                        prompt_template: str,
                                        num_data_gens_per_txt_files: int = 100,
                                        batch_size: Optional[int] = sys.maxsize,
                                        batched: Optional[bool] =True,
                                        use_mathematica_code: bool = True,
                                ) -> list[dict]:
    """
    Creates a JSON Lines file from .txt files containing math problems across directories.

    Args:
        source_path (str): Path to the source directory containing folders of .txt files.
        output_path (str): Path to save the .jsonl output file.

    Returns:
        List[Dict[str, str]]: A list of dictionaries representing the json lines.
    """
    source_path: Path = Path(source_path).expanduser()
    output_path: Path = Path(output_path).expanduser()
    print(f'{source_path=} {output_path=}')

    # -- Collect all the raw data
    print()
    all_raw_data: list[dict] = []
    for root, dirs, files in os.walk(source_path):
        # print(f'\n{root=} \n{dirs[:3]=} \n{files[:3]=}')
        for f in files[:num_data_gens_per_txt_files]:  # Select up to 50 .txt files per directory
            if f.endswith('.txt'):
                # print(f'\n{root=} \n{dirs[:3]=} \n{files[:3]=}')
                # selected_files = [os.path.join(root, f) for f in files if f.endswith('.txt')][:num_data_gens_per_txt_files]
                file_path = os.path.join(root, f)
                category: str = root.split('/')[-2]
                problem_type: str = root.split('/')[-1]
                mathematica_script_name: str = f'{problem_type}.nb'
                # extract problem, answer, category, prob_type, (empty) solution
                problem_data: dict = extract_problem_data(file_path)
                all_raw_data.append(problem_data)
                problem_data['problem_type'] = ' '.join(problem_type.split('_'))  # "variance_and_std" -> "variance and std"
                problem_data['category'] = ' '.join(category.split('_'))  # "counting_and_statics" -> "counting and statics"
                problem_data['mathematica_script_name'] = mathematica_script_name
                # get everything up to the last but last is not included
                path_2_mathematica_scripts: str = '/'.join(root.split('/')[:-1])  # <path>/<cat> since cat has the nb files
                mathematica_script: str = open(f'{path_2_mathematica_scripts}/{mathematica_script_name}', 'r').read()
                print(f'{problem_data=}')
                problem_data['mathematica_script'] = mathematica_script
                print(f'{problem_data=}')
                print()
    print(f'{len(all_raw_data)=}')

    # TODO -- Batch the data
    if batched:
        from evals.utils import batch_data
        assert batch_size > 0, f'batch_size should be greater than 0 but got: {batch_size=}'
        all_raw_data: list[dict] = batch_data(all_raw_data, batch_size=batch_size)
        num_batches: int = len(all_raw_data)
        print(f'len(all_raw_data)/batch_size = {len(all_raw_data)}/{batch_size} = {math.ceil(len(all_raw_data)/batch_size)}')
        print(f'{num_batches=}')

    # -- Gen synthetic data
    from evals.inference_eval import VllmGenerator, OpenAIGenerator
    print(f'{type(all_raw_data)=} {type(all_raw_data[0])=}')  # to see which type of prompts we will make
    if isinstance(gen, VllmGenerator) and isinstance(all_raw_data[0], dict):  # list of single string prompt
        all_raw_data: list[dict]
        # - Generate synthetic solution string for each problem and answer (using meta-data too)
        all_data_with_gen_synth_soln: list[dict] = []
        for raw_data in tqdm(all_raw_data, total=len(all_raw_data)):
            # Make prompt to generate synthetic solution
            problem: str = raw_data['problem']
            answer: str = raw_data['answer']
            category: str = raw_data['category']
            prob_type: str = raw_data['problem_type']
            mathematica_code: str = raw_data['mathematica_script']
            # prompt: str = prompt_template.replace('{problem}', problem).replace('{answer}', answer).replace('{category}', category).replace('{prob_type}', prob_type).replace('{mathematica_code}', mathematica_code)
            prompt: str = prompt_template.replace('{problem}', problem).replace('{answer}', answer).replace('{category}', category).replace('{prob_type}', prob_type)
            # Generate synthetic solutions (completions), n>1 means same reasoning per solution but phrased differently
            request_outputs_per_batch_prompts: list[RequestOutput] = gen.llm.generate(prompt, gen.sampling_params)
            completions_per_prompt: list[CompletionOutput] = request_outputs_per_batch_prompts[0].outputs
            synthetic_solutions: list[str] = [completion.text for completion in completions_per_prompt]
            # Create dict data point per problem answer pair with its synthetic solutions
            print(f'First solution: {synthetic_solutions[0]=}')
            print(f'Real answer: {answer=}')
            if len(synthetic_solutions) == 1:
                raw_data['solution'] = synthetic_solutions[0]
                all_data_with_gen_synth_soln.append(raw_data)
            else:
                data_points_per_ans: list[dict] = [{**raw_data, 'solution': solution} for solution in synthetic_solutions]
                all_data_with_gen_synth_soln.extend(data_points_per_ans)
                assert len(synthetic_solutions) == gen.sampling_params.n, f'{len(synthetic_solutions)=} != {gen.sampling_params.n=}'
        assert len(all_data_with_gen_synth_soln) == len(all_raw_data) * len(synthetic_solutions), f'{len(all_data_with_gen_synth_soln)=} != {len(all_raw_data) * len(synthetic_solutions)=}'
    elif isinstance(gen, VllmGenerator) and isinstance(all_raw_data[0], list):  # list of batch of prompts
        all_raw_data: list[list[dict]]
        # - Generate synthetic solution string for each problem and answer (using meta-data too)
        all_data_with_gen_synth_soln: list[dict] = []
        for batch_probs_ans in all_raw_data:
            batch_prompts: list[str] = [prompt_template.replace('{problem}', prob_ans['problem']).replace('{answer}', prob_ans['answer']).replace('{category}', prob_ans['category']).replace('{prob_type}', prob_ans['problem_type']) for prob_ans in batch_probs_ans]
            request_outputs: list[RequestOutput] = gen.llm.generate(batch_prompts, gen.sampling_params)
            for request_output_per_prompt in request_outputs:
                completions_per_prompt: list[CompletionOutput] = request_output_per_prompt.outputs
                synthetic_solutions: list[str] = [completion.text for completion in completions_per_prompt]
                # Create dict data point per problem answer pair with its synthetic solutions
                for i, prob_ans in enumerate(batch_probs_ans):
                    prob_ans['solution'] = synthetic_solutions[i]
                    all_data_with_gen_synth_soln.append(prob_ans)
        assert len(synthetic_solutions) == gen.sampling_params.n, f'{len(synthetic_solutions)=} != {gen.sampling_params.n=}'
        assert len(all_data_with_gen_synth_soln) == len(all_raw_data) * len(synthetic_solutions), f'{len(all_data_with_gen_synth_soln)=} != {len(all_raw_data) * len(synthetic_solutions)=}'
        raise NotImplemented
    elif isinstance(gen, OpenAIGenerator):
        raise ValueError(f'Invalid value for {gen=}.')
    else:
        raise ValueError(f'Invalid value for {gen=}.')
    print(f'{len(all_data_with_gen_synth_soln)=} {len(all_raw_data)=}')
    assert isinstance(all_data_with_gen_synth_soln, list) and isinstance(all_data_with_gen_synth_soln[0], dict), f'{type(all_data_with_gen_synth_soln)=} {type(all_data_with_gen_synth_soln[0])=}'

    # -- Save all data to a jsonlines file
    output_path: Path = output_path.expanduser()
    output_path.parent.mkdir(parents=True, exist_ok=True) if output_path.is_dir() else output_path.parent.mkdir(parents=True, exist_ok=True)
    print(f'{output_path=}')
    with open(output_path, 'w', encoding='utf-8') as file:
        for synth_data in all_data_with_gen_synth_soln:
            json.dump(synth_data, file)
            file.write('\n')
    return all_data_with_gen_synth_soln

# -- Main

def main_gen_synth_data(
        # model: str = 'DEFAULT',  # e.g., Mistral-7B-Instrcut-v0.2 on http://<IP_ADDRESS>:12345
        # model: str = 'deepseek-ai/deepseek-math-7b-instruct',
        model: str = 'mistralai/Mistral-7B-Instruct-v0.1',
        # model: str = 'gpt2',
        n: int = 4, # num seqs to return for given prompt
        max_tokens: int = 2048,
        # max_tokens: int = 4096,
        top_p: float = 0.95,
        temperature: float = 0.8,
        num_beams: int = None,
        best_of: int = None,
        use_beam_search: bool = False,
        num_similar_data_pts_txt_files: int = 30,  # number of problems to use (that were generate from the same mathamatica script)
    ):
    """ Gen synthetic data from math problems. """
    from evals.prompts_evals import STOP_TOKENS
    from evals.inference_eval import VllmGenerator, OpenAIGenerator
    from evals.utils import get_dtype_for_vllm
    from evals.inference_eval import VllmGenerator
    # -- Print
    print(f'{model=}')
    print()

    # -- Get vllm generator
    prompt_template: str = MATHEMATICA_SYNTH_GEN_PROMPT_TEMPLATE
    stop: list[str] = STOP_TOKENS
    sampling_params: SamplingParams = SamplingParams(n=n, max_tokens=max_tokens, top_p=top_p, temperature=temperature, stop=stop, use_beam_search=use_beam_search, best_of=best_of)
    print(f'{model=}, \n{sampling_params=} \n{num_similar_data_pts_txt_files=}')
    # gen: OpenAIGenerator = OpenAIGenerator(model=None, sampling_params=sampling_params)
    dtype = 'bfloat16' if torch.cuda.is_bf16_supported() else 'float16'
    llm: LLM = LLM(model=model, dtype=dtype)
    gen: VllmGenerator = VllmGenerator(llm, sampling_params)
    print(f'{sampling_params} \n {sampling_params=}')

    # -- Generate synthetic data & save it as a jsonlines file
    path_2_src_dataset: str = '~/data/amps/mathematica/'
    output_path: str = '~/gold-ai-olympiad/data/amps/mathematica/train.jsonl'
    mathematica, num_data_gens_per_txt_files = False, 2
    lst: list[dict] = gen_data_set_from_ans_str_2_jsonl(gen, path_2_src_dataset, output_path, prompt_template, num_data_gens_per_txt_files=num_data_gens_per_txt_files)
    print(f'Number of jsonlines: {len(lst)=}, written to {output_path=}, from {path_2_src_dataset=}')

if __name__ == '__main__':
    import time
    start = time.time()
    # _test_batch_data()
    # main_math_jsonlines_file()
    main_gen_synth_data()
    print(f"Done!\a Time: {time.time()-start:.2f} sec, {(time.time()-start)/60:.2f} min, {(time.time()-start)/3600:.2f} hr\a")

ref: https://stackoverflow.com/questions/78487360/assertion-with-no-scription-in-vllm-with-deepseekmath-7b-model-why-how-to-fix

Help?
On Tue, Nov 26, 2024, 6:08 PM github-actions[bot] @.***>
wrote:

Closed #4849 https://github.com/vllm-project/vllm/issues/4849 as not
planned.
—
Reply to this email directly, view it on GitHub
https://github.com/vllm-project/vllm/issues/4849#event-15445196485, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOE6LQ7GRLLDGPSKAWYF632CUSTDAVCNFSM6AAAAABHZJJFPKVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJVGQ2DKMJZGY2DQNI
.
You are receiving this because you authored the thread.Message ID:
@.***>


Do not add useless quotes

Exemple:

Could generate: (without quotes)

Hi @shadawcraw , no it is not fixed yet


Add "rocm/include/roctracer" to the list of system include dirs.

Building TF with ROCm3.5-rc1 runs into the following error
Execution platform: @local_execution_config_platform//:platform
In file included from ./tensorflow/core/profiler/internal/gpu/rocm_tracer.h:23:0,
                 from tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:16:
bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/roctracer/roctracer_hcc.h:43:23: fatal error: roctracer.h: No such file or directory
compilation terminated.
Target //tensorflow/tools/pip_package:build_pip_package failed to build

The error is caused because the following change in the roctracer_hcc.h file
ROCm 3.3 version : #include "roctracer.h"
ROCm 3.5 version : #include <roctracer.h>
In order to accomodate this change, we need to add "rocm/include/roctracer" to list of system include dirs. This is not the ideal fix (think that would be to revert the change in roctracer), but will have to do for now

/cc @sunway513 @whchung @jerryyin @ekuznetsov139

Can one of the admins verify this patch?

LGTM


404 when attempting to 'promote' or 'delete' a 'potential finding'

Hi All,
Running the Release Mode 2.2.0 deployed with Docker-Compose.
Experiencing a 404 when trying to promote or delete a potential finding under a test.
Steps to reproduce the behavior:

Open an active Test (Product > Engagement > Test)
Add Potential Findings using the TextField 'Add a potential finding...' and click the button '+ Add Potential Finding'
Click the button 'Promote To Finding' or 'Delete' under actions for the Potential Finding created in step 2.
Page returns a 404 Not Found

Expecting the Promote to Findings form to be loaded.
Logs are showing:
[10/Sep/2021 03:21:52] WARNING [django.request:224] Not Found: /stub_finding/40/promote
WARNING:django.request:Not Found: /stub_finding/40/promote
Reviewed the finding/urls.py and sighted a match however it doesn't seem to be matching. If I copied both Promote and Delete patterns above the Add, the request does work but with a delay.
Screenshot of the modified finding/urls.py pattern. (copied promote and delete above the add pattern)

In the latest dev I can't even add a potential finding, the add already gives a 404 for me.

Maybe we should add an integration test and also move to path based url routing.
https://stackoverflow.com/questions/47947673/is-it-better-to-use-path-or-url-in-urls-py-for-django-2-0

I take it

fixed in dev / 2.3.0

Thank you


About LeftWindowCommands and RightWindowCommands

MetroWindow.LeftWindowCommands/RightWindowCommands is disabled.How can I solve it?

@ftgithub What do yo mean with disabled? I don't see your screenshots...


Release latest

Could you do a release with latest changes please?
Thanks in advance

Sure! Will do!

https://github.com/ashcrow/filetranspiler/releases/tag/1.1.2

Thanks!


[5.6] API Resource wrapper bypass

ResourceResponse doesn't wraps response data if there is a model property with same name as wrapper keyword.

There's an edge case here that is still left untreated, when the model only has only has one property and it is named the same as the wrapper keyword.
I reckon that's a silly use case and possibly an indicator of user error, but it should be documented.

i address that issue when model has only one property, on other hand what's the point having model with one property.
i totally agree that detection if response is wrapped should be done in complete different way. for whatever reasons you would like to have same names for property and wrapper, wrapper should have nothing to do with model properties.

I am closing this pull request because it lacks sufficient explanation, tests, or both. It is difficult for us to merge pull requests without these things because the change may introduce breaking changes to the framework.
Feel free to re-submit your change with a thorough explanation of the feature and tests - integration tests are preferred over unit tests. Please include it's benefit to end users; the reasons it does not break any existing features; how it makes building web applications easier, etc.
Thanks!


Add NASA Mars, Moon and Vesta tile providers

https://api.nasa.gov/#:~:text=Vesta/Moon/Mars Trek WMTS are web map tile services which let you get this stuff.  We should add them to TileProviders!  It should be simple to implement as an example for the docs as well.

Yeah would be cool! But what is the projection even??
It would be fun to plot onto a mesh so we can make them globes, for earth too really...
That should be doable?

And also they are not XYZ tiles? maybe we need a way to distinguish that from the provider type. I don't think they would work in Leaflet.jl or Tyler.jl without a bunch of work.
http://{WMTS endpoint}/1.0.0/{Style}/{TileMatrixSet}/{TileMatrix}/{TileRow}/{TileCol}.png

So I think Tyler already plots onto meshes - we would just have to densely sample them and define some kind of mapping.
Is there a particular reason that Tyler needs to have scene data?  If so we could make it at least backed by an ordinary plot recipe, and then it could even have a transformation from e.g 2d to 3d (sphere).
also: isn't this almost exactly the XYZ tile definition?  Z is tile matrix, x and y are tilecol and tilerow (or so it seemed to me at least).

Ahh yeah I just misunderstood the two matrix variables. It works fine:
using TileProviders
mars = Provider("https://api.nasa.gov/mars-wmts/catalog/Mars_Viking_MDIM21_ClrMosaic_global_232m/1.0.0//default/default028mm/{z}/{y}/{x}.jpg")
Tyler.Map(Extent(; X=(0, 10), Y=(0, 10)); provider=mars)

Currently, Tyler is plotting images directly onto a mesh using UV coordinates
https://github.com/MakieOrg/Tyler.jl/blob/346b2c78234374c7493c94c229804570fe65844f/src/Tyler.jl#L175-L185
if we just subdivide this, for which the code is actually already there, and apply a transformation from latlong to an ECEF cartesian coordinate system (see Geodesy.jl), then it should "just work"!
When you consider Makie, everything is actually 3d - 2d plots are simply a special camera setting.
I have to finish up some slides for a talk this evening but can give this a look after that.


WIP: update handler for tokens

This provides an update route for tokens, but it needs some RBAC-fu

Hi @maxkorp. I'm stomping old PRs 😱


Fix the issue with forking repo

Pull Request Test Coverage Report for Build 1304

6 of 7 (85.71%) changed or added relevant lines in 2 files are covered.
40 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.3%) to 54.193%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

repairnator/repairnator-pipeline/src/main/java/fr/inria/spirals/repairnator/process/git/GitHelper.java
4
5
80.0%

Files with Coverage Reduction
New Missed Lines
%

repairnator/repairnator-core/src/main/java/fr/inria/spirals/repairnator/config/RepairnatorConfig.java
40
0.0%

Totals

Change from base Build 1298:
0.3%

Covered Lines:
1913

Relevant Lines:
3530

💛  - Coveralls


Fix install instructions for 0.4+

I'm not sure if the plan is merging back to sehrope's repo, but in the meantime this will help people avoid the same mistake I just made :)

my intent is not to replace his plugin....but i'm not sure where he stands on maintenance....


Update Istio support to beta phase

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?:

NONE

🚀
/approve


replace example figure

Given what we've learned from Franco and Josiah's presentation the other day, we could probably replace the figure on the front of this app with something running non-ensemble tractography and thereby have a more visually appealing example.

By the front of this app, do you mean this?

https://brainlife.io/ui/tractview/demo.html

I was thinking this: https://github.com/brainlife/app-wmaSeg/blob/master/wmaSeg.png

@DanNBullock can you please make a figure that you like? We will need a PNG

I'll try and generate one as I test the refactored version of the segmentation


Add Question Answering Node

This PR should  be stacked on https://github.com/PTG-Kitware/angel_system/pull/297.

This PR should be blocked on https://github.com/PTG-Kitware/angel_system/pull/312


Rename radioConfig.py to radioConfig.py.example

Currently if you edit radioConfig.py to configure your location git will want to track those changes, and should the config file change in future you'll need to resolve that conflict.
It would be cleaner if the file in git was renamed to be radioConfig.py.example so that sites can copy it to be radioConfig.py and then modify it to suite.  Then when changes happen they can merge them by hand without disturbing the state of the git repository locally.
How does that sound?
All the best!
Chris

Hi Mario,
That's great, your choice of name is a better idea!
All the best,
Chris


UI: Implement viewer for in-package PackageVariantSet resource

Implement UI resource editors for PackageVariantSet.
Unit test coverage.
Parent issue

/retitle Implement viewer for in-package PackageVariantSet resource

/retitle UI: Implement viewer for in-package PackageVariantSet resource

Created a fake PVS yaml here for DEV purpose:
https://github.com/ganchandrasekaran/nephio-example-packages-ui-test/blob/main/free5gc-smf-org-example/packagevariantset.yaml

Here is a Viewer implementation for PVS,
@johnbelamaric @ChristopherFry please let know if any changes are need?

@ChristopherFry  @johnbelamaric  Here is PR for review: https://github.com/nephio-project/kpt-backstage-plugins/pull/3


[TF 1.14 Keras] probably bug in network._map_graph_network

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Both OS-X and Ubuntu
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): 1.14.0 binary
TensorFlow version (use command below): 1.14
Python version: 3.6.
Bazel version (if compiling from source): NA
GCC/Compiler version (if compiling from source): NA
CUDA/cuDNN version: NA
GPU model and memory: NA

Describe the current behavior
I have a CNN model that works well with tensorflow==1.13.1 (building the network, training the network and predicting from the network), but after the update to tensorflow==1.14.0 I'm getting following exception:
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("dropout_1/cond/Merge:0", shape=(?, 160, 160, 16), dtype=float32) at layer "concatenate". The following previous layers were accessed without issue: ['net_input', 'initial_conv2D', 'batch_normalization', 'activation', 'conv2d', 'dropout']

Other info / logs
After debugging and going through the implementation of network._map_graph_network it seems to me that there is either some bug (significantly changed behaviour between tensorflow==1.13.1 and tensorflow 1.14.0) in the graph disconnections checking.
When I run the code with tensorflow==1.13.1 the tensorboard graph looks as follows:

From this I'm not sure why layer dropout_1 (conditional dropout) is checked in the concatenate layer? I assume that droupout should have been checked there?

Thank you for the report. Just a small clarifying question: can you also try against nightly version? I think this might have been fixed already, but I might be wrong.

@mihaimaruseac I've tried it against tf-nightly and I'm getting exactly the same exception.

I'll defer to the Keras team then, as I'm not sure where to go from here

same problem here too

@sedghi interesting, thank you, but unfortunately in my case workaround using tf.concat, at least as Lambda layer does not work for me. I'm still getting the same error message.

very much related to https://github.com/tensorflow/tensorflow/issues/30355

I'm further proceeding in debugging. It currently looks like in the case when layer is an input to more than one concatenation than following strange thing happen:
My testing graph (NOTE: tensorboard printed from tensorflow==1.13.2)

For some reason concatenate layer in the red circle has correctly 2 input layer, but incorrectly 3 input tensors:

I've noticed both of this in _map_graph_network and inside in build_map function. From this I assume that the bug is not in network._map_graph_network function, but somewhere in the construction of the graph.
Does anybody has some idea where should I look further? It seems to me that there might be some problem in the keras.engine.Layer keras.layer.merge._Merge or maybe somewhere else? Any suggestions @mihaimaruseac @fchollet

Hi There,
We are checking to see if you still need help on this, as you are using an older version of tensorflow which is officially considered end of life . We recommend that you upgrade to the latest 2.x version and let us know if the issue still persists in newer versions. Please open a new issue for any help you need against 2.x, and we will get you the right help.
This issue will be closed automatically 7 days from now. If you still need help with this issue, please provide us with more information.


MP3's are not branded

MP3's don't contain metadata, showing 'unknown artist'. Don't know about other file formats.

Fixed. Thanks for reporting.


ionic 5 (cordova-android 12,targetSdkVersion-33,compileSdkVersion-24) some android devices not token generate

I have the same issue.

Look like you are invoking the plugin before the deviceready event has fired.
Please build and run the example project as a know working codebase for reference.
If you are able to reproduce the issue in the example project, please open a new issue and fill out the issue template in full.


fix issue with params.pp with strict_variables enabled

With strict_variables enabled Puppet runs will fail if a variable is referenced but is not defined. The error you receive is Evaluation Error: Unknown variable:.
https://docs.puppetlabs.com/references/latest/configuration.html#strictvariables
There are several chances for this to happen because of the conditional variable definitions in params.pp. This pull request ensures all referenced variables are defined in params.pp.

@eoly thanks, and apologies it's taken me an age to get round to looking at this.


[WIP] expose storage_cluster to automation

@agrare @gmcculloug @bzwei @blomquisg Please take a quick look if this makes sense. thanks

@miq-bot label wip

@jameswnl This looks good based on PR #6706.  @agrare Will this class eventually have relationships to/from other models?  When those are added we will need to reflect that here as well.

@jameswnl LGTM :+1:
@gmcculloug A StorageCluster will have 0..N Storage children.  Not sure if that has to be represented in this model or if it will "Just Work" :tm:

If it is an active_record relationship then it would need to be defined using something like:
expose :storages, :association => true
But is should only be defined if it is already defined at the active_record layer which currently it is not.
At this point the service model matches the AR model (they both just derive from EmsFolder) so you are good for now.

Thanks guys for your help in my first PR!


Unhandled message type: {"header":{"tag":"h3",...

It looks like Groups pages choke on header lines other than h1 maybe?

App: Groups
Version: undefined
Vere Version:
Hash: undefined
Synced From: undefined
Location: /apps/groups/groups/~magbel/in-tenebris-silva/channels/diary/~magbel/scholia-cotidie--daily-notes-/note/170141184506537112992258515129405450223
Error: Unhandled message type: {"header":{"tag":"h3","content":["Coexistence under contemporary niche theory"]}}
  at Hn (http://<IP_ADDRESS>:8081/apps/groups/assets/index-d62c0078.js:139:48323)
  at Mu (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:30:19470)
  at Bp (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:32:43828)
  at Tp (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:32:39610)
  at gh (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:32:39538)
  at So (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:32:39393)
  at ql (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:32:35786)
  at kp (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:32:34737)
  at x (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:17:1561)
  at MessagePort.oe (http://<IP_ADDRESS>:8081/apps/groups/assets/react-beautiful-dnd-5ffc626c.js:17:1928)

Hey, fyi, I was unable to repro this. Is it still an issue for you?

I'd have to see if I can repro the original Markdown.  (I altered it to make it work.)  So we can close now.