Face Preparation for Model Prediction

Hi,
How can I read/align/prepare face for making predict on trained model?
I made code like this:
    model = create_deepface()
    model.load_weights('VGGFace2_DeepFace_weights_val-0.9034.h5')
    img = load_img(path)
    img = img.resize((152,152))
    img_array = img_to_array(img)
    img_array = img_array.reshape((-1, 152, 152, 3))
    res = model.predict(img_array)

I get different results for every face belonging to the same person.
Thanks a lot for a help

You need to use the preprocess_image function which is an essential part of the train and test data pipeline. In your case the function can be called in the following way.
# required imports
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from deepface import create_deepface
from deepface import Dataset

model = create_deepface()
model.load_weights('VGGFace2_DeepFace_weights_val-0.9034.h5')

# fine tune the deepface model for your dataset
# for fine tuning, remove last dense layer only; (as per paper)
# set all pretrained parameters to non trainable
# add a dense layer with dim = num_classes (your dataset)
# set dense layer to trainable and train the model

img = load_img(path)
img = img.resize((152,152))
img_array = img_to_array(img)
img_array = img_array.reshape((-1, 152, 152, 3))
res = model.predict(img_array)

If there are any issues regarding the fine tuning step, please do let me know.

For training the DeepFace network, 2D aligned (2D frontalisation using 5 fiducial points similar to the one specified in paper) images from the VGGFace2 dataset have been used. It is expected that we use the same alignment technique (2D alignment of tight face crops) to make accurate predictions using DeepFace feature vectors.
Shortly, will update the repository with the code required to obtain 2D aligned tight face crops from images in dataset. 2D-alignment in our case had been achieved using dlib.get_face_chips method using shape_predictor_5_face_landmarks.
Currently, I'm working on the 3D frontalisation step to replicate the exact 3D alignment step as specified in paper. Hope to add include those changes and train the network using 3D frontalised images soon.

The code for the 2D face alignment seems fine.
The only limitation being that the model.predict method can make accurate predictions only after the model has been fine tuned (last dense layer only) on your required dataset.
The original implementation (which actually represents the pretrained weights) makes use of a softmax classifier trained on 8631 classes from the VGGFace2 dataset. So, it is unlikely to make predictions the same class label unless the face crop being used belongs to one of those classes.
DeepFace doesn't use a triplet loss based implementation unlike FaceNet by Scroff et al. which can be directly used for a face verification task. Instead, the DeepFace feature vector obtained from the second last dense layer from the network can be used to learn a face verification problem.

Thank you
Now I have code like this:
from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import array_to_img
import numpy as np
import dlib;
from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image

detector = dlib.get_frontal_face_detector()
sp = dlib.shape_predictor('shape_predictor_5_face_landmarks.dat')

def processdeepface(path):
    img = dlib.load_rgb_image(path)
    dets = detector(img, 1)
    for k, d in enumerate(dets):
        shape = sp(img, d)
        face_chip = dlib.get_face_chip(img, shape, size=152)
        image = preprocess_image(face_chip)
        img_array = img_to_array(image)
        img_array = img_array.reshape((1, IMAGE_SIZE[0], IMAGE_SIZE[1], 3))
        res = model.predict(img_array)

Am I doing it in proper way?
Faces belonging to the same person should have the same output up? Am I correct?
Thanks

@swghosh @lukkowal5 Hi,how can i process the 8631 output to a image? Thanks.

Sorry for the delay in reply.

@ZHUANG-JLU,
What is the actual problem that you're motivated to solve?
Is it a face verification task (relating to matching a pair of images whether they belong to same person or not) or a face recognition task (relating to a multi class classification problem with some k number of people/identities in dataset)?
Please confirm about the same then I can get back to you with some code that can perform either based on your requirement.

@ZHUANG-JLU,
What is the actual problem that you're motivated to solve?
Is it a face verification task (relating to matching a pair of images whether they belong to same person or not) or a face recognition task (relating to a multi class classification problem with some k number of people/identities in dataset)?
FYI, the last layer of the CNN i.e. the F8 layer is trained to classify samples of the VGGFace2 dataset in this case. The pre-final layer, F7 layer with 4096 dimensions is used to extract the DeepFace feature vector. The simplest workaround for a face recognition problem is to replace the F8 layer with a new dense layer of dimension equal to number of classes in new dataset, and retrain the network for a few epochs (eg. 20) with all weights frozen upto F7. [some similar work to this is available here: FineTuning DeepFace on a smaller dataset.ipynb]
Please confirm about the same then I can get back to you with some code that can perform either based on your requirement.

Face Detection/Recognition/Alignments.Thanks for your reply.

@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb]

@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb
I will see it.Thanks.

Hey there, I am new in this field can you tell me how to run this code.

Also is there any way I can run it on my custom dataset.

@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb

Hi @thepranaygaur,
Can you refer to the notebook provided in the link to run the face recognition model on a custom dataset?
Assuming that you've a directory containing pictures of faces of people (tight face crops, aligned) segregated into sub directories basis their class labels / identity of the person, you can use the notebook for end to end code.
Do let me know if you have any queries.
For face detection and alignment please use dlib.get_frontal_face_detector and dlib.get_face_chips respectively as given in https://github.com/swghosh/DeepFace/issues/1#issuecomment-555579591

Hello, thanks for your work first.
I am reading DeepFace, and I am quite confused about the 3D alignment step. To be specific, I cannot tell what is the affine 3D-to-2D camera P (what are the 8 unknowns mean?). Please help me. Thank you a lot.

Hi @FunkyKoki, thanks for reaching out!
In my repository, the DeepFace model which have been trained on VGGFace2 dataset uses tight face crops with 2D alignment only (atleast, as of now) using dlib.get_face_chips (works reasonably well under various poses, except poses that lack frontal face which is the reason I had remove from training the face pictures that lacked frontal poses). I went through the 3D alignment process as discussed in the original paper but failed to be able to replicate the same work primarily due to lack of availability public 3D face datasets and level of difficulty associated with it.
The methodology discussed uses 3D to 2D mapping of face keypoints with the help of an affine camera (not really sure how that can be done with regular 2D face crops) and a trained 3D face model. As a starting point, I'd recommend you get started with these resources which should help you develop more clarity on this topic.

Viewing Real-World Faces in 3D (https://talhassner.github.io/home/publication/2013_ICCV)
face-frontalization repo by dougsouza (https://github.com/dougsouza/face-frontalization) - this is a Python port of the original MATLAB implementation of the paper Effective Face Frontalization in Unconstrained Images (https://talhassner.github.io/home/publication/2015_CVPR_1)
Both works cited here include work done by Tal Hassner, who has relevant work in this field (work cited in DeepFace paper).
Introductory details about camera models and 3D affine cameras (https://www-users.cs.umn.edu/~hspark/CSci5980/CameraModel1.pdf)

Let me know if you are willing to collaborate with me on this at a suitable time / looking forward to contributing some code that is related to 3D face frontalization. Will be more than happy to be of any help.
Good luck!

Thank you so much for your sharing.
Indeed, I have 2 questions about the 3D-to-2D process.
First, what are the 8 unknows in the affine camera P mean? Location, gesture and lens?
Second, the author added the residual r into the 3d coordinates, but this operation can be done only if the coordinates systems are aligned, how could that be done?
I would check the resources you shared and try to re-implement the whole process. By the way, actually, in the modern network, the strong 3D alignment seems not to be a necessity. I am interested in the 3D alignment only because that I am working in the 3D reconstruction of face.
I would let you know when I solve the two questions above.
Thanks  again.

@swghosh Hello, I'm trying to use https://github.com/serengil/deepface deepFace facial attribute analysis, but the performance toward certain age group is bad. So I'm looking for a way to train/ tune the age/gender/race classifier with my own dataset. I'm wondering if this is possible to do? I saw this repo is using VGG face for pretrain weight. Can I retrain with my  dataset? May you give me some insight? Thank you!

Hey @zzzzmoya,
That sounds like an interesting concept. Indeed, the original VGG-Face2 dataset focuses on a detailed study about various facial attributes. I'm not sure about what serengil/deepface offers with regard to Facial Attribute Analysis. I'm skeptical that the original residual architecture mentioned (if I remember correctly) in the VGGFace2 paper should help you with get a head start on your work. And, I'd assume that the VGGFace2 dataset is inclusive enough irrespective of age groups so you can train your own model with it and later tune it on your dataset. Feel free to ping me if there's related additional discussion to be done on this regard (preferably, by creating a separate issue).

Hey @swghosh, awesome post! I have a question:
Could the classification task be repurposed to say facial analysis? Eg: Using the same data hierarchical structure, what if instead of matching (one to many) subjects, the labels would be race-ethnicity identification where folders would be labeled into a limited number of categories?
From your reply to @ZHUANG-JLU, I'm thinking of creating a new F8 layer with the number of categories for my task and train the model first as a classifier (i.e. training just the F8 layer) using VGGFace2 pre-trained weights and then fine-tuning setting a portion of the DeepFace model to trainable. Curious to know your thoughts? Would you by any chance have any sample code for creating an additional F8 layer? Thanks!

Hey @swghosh, awesome post! I have a question:
Could the classification task be repurposed to say facial analysis? Eg: Using the same data hierarchical structure, what if instead of matching (one to many) subjects, the labels would be race-ethnicity identification where folders would be labeled into a limited number of categories?
From your reply to @ZHUANG-JLU, I'm thinking of creating a new F8 layer with the number of categories for my task and train the model first as a classifier (i.e. training just the F8 layer) using VGGFace2 pre-trained weights and then fine-tuning setting a portion of the DeepFace model to trainable. Curious to know your thoughts? Would you by any chance have any sample code for creating an additional F8 layer? Thanks!

I am certain that the output of F7 layer can be used as features for other facial related tasks as well, Facial Analysis in your case. Yes, there are quite a lot of chances that adding a new layer on top F7 or even experimenting the same with the output of L6 and then stacking custom Dense layers should help you fine-tune the model on a new facial task with new categories.
In case you want a model that performs generic facial analysis, you should be able to do that using a MTCNN (multi-task CNN) which would provide for multi-label classification as well as regression tasks under a common end-to-end neural network.
For just race/ethnicity classification, I'd hope that output of features from either F7 or L6 (can try experimenting with either: F7 preferred) should suffice given there is one or more Dense layer(s) stacked on top (subject to hyper-parameter tuning and experiments).

Would this notebook not be sufficient for you to guide you on training a new face model with new classification categories? (in your cases, categories would be race/ethnicity labels; everything else code-wise should be the same).
FineTuning DeepFace on a smaller dataset.ipynb
You could also refer to any resources where generic fine-tuning tasks using Keras models have been described for that purpose. Should be pretty easy to load the weights and just remove the last layer to use your own.
As an extra, if you'd like to fine-tune the complete model (inclusive of Conv layers, Locally-connected layers) but please note that it is extremely computationally expensive and would either need a GPU cluster or a TPU. (Google Colab provides for free TPUs which should suffice)

hi ,
i used FineTuning DeepFace on a smaller dataset.ipynb with my dataset but i can't get good accuracy and it give me just 10%
thnx for help

Hi @saidafef: Apologies for the delayed response.
You could probably check for any red flags in any of the following aspects:

Pre-processing is same as that used in training or not
The size of the dataset and number of samples per class should be reasonable as the DeepFace feature classifier would perform only as good as the number of samples it is trained on for each class (unlike one-shot face verification tasks like FaceNet)
Occlusions / necessary variations among your training and validation datasets
Learning rate can be tuned significantly along with dropout value in range 0.2-0.5 for some performance improvements
Full fine tuning with a very low exponentially decayed learning rate? (only you have cumbersome resources available because training the locally connected layers is too much resource hungry)

Feel free to ping with any additional details.

Hi team, I am looking for face verification problem (given two images identify if they are of the same individual or of different people). Do you have a similar implementation like this (https://colab.research.google.com/drive/1otl-4tJaCfyCJF33bvnjzeuUdldcWysA) for my case. Fine tune the model to identify images of same people in a given dataset.


Base64 encoder should support the URL-safe alphabet

Currently, the Base64 encoder in the core library will reject any Base64 data that includes the - or _ characters, or uses %3D as a padding character. Although technically "base64 with URL and filename safe alphabet" is a different encoding, it's likely that users won't realize this or won't know how to distinguish between them and will provide URL-safe base64 to the main encoder. This is particularly likely because the previous standard decoder in package:crypto does accept this alphabet.
I'm marking this as 1.13 since the base64 converter is a 1.13 feature.

I don't want the core version of base64 to have too many bells and whistles. I'd rather add an extended base64 codec in package:convert.

I understand, and I think encoding a URL-safe alphabet should go in package:convert. But users will expect that the decoder will decode all valid base64. If the input comes from external sources, they may not even be able to control whether it's URL-safe or not. Violating that assumption and rejecting the input will introduce subtle brokenness and incompatibility.

@lrhn, is that something we can address in 1.13?

That depends on how much time we have until 1.13.
It's pretty easy to make the decoder handle _ and - as input, but not so easy to accept percent-encoded data.
Not supporting %3D as padding was a deliberate choice - percent encoding is not part of base-64 at all, it's something that happens when you embed base-64 encoded data in a URI, and it should be undone when you extract the data again (and the DataUri class would do that). Putting it into the base-64 codec is bad separation.

@lrhn That's arguable. According to the spec:

The pad character "=" is typically percent-encoded when used in an URI, but if the data length is known implicitly, this can be avoided by skipping the padding; see section 3.2.

Percent-encoding is certainly part of the spec, although what mandate that holds for an implementation is ambiguous. I'd be okay if it wasn't supported by the decoder as long as _ and - are.

@nex3 Is this addressed with https://codereview.chromium.org/1419983003/ ?
@lrhn can we get this landed and get a merge request in?

Is this addressed with https://codereview.chromium.org/1419983003/ ?

Yep!

Merge request to get this into 1.13 - https://github.com/dart-lang/sdk/issues/24726

It ended up supporting %3D as padding, but no other escapes, and it still does not allow non-alphabet characters inside the encoded data (not even newlines).

Landed and merged into 1.13


recover test case for "encode/bytes"

pending on cadl-ranch fix PR https://github.com/Azure/cadl-ranch/pull/455

move to https://github.com/Azure/autorest.python/pull/2293


InceptionResNetV2

Well it can run, but I'm going to have to leave it overnight

How are you training it?

So its weights are pretrained on the imagenet dataset. Basically all I have to do it to retrain the first few and last few layers to adapt them to the new classes and input dimensions. Their weights are the ones that have to be retrained the most. Then I let all the weights become trainable, but i'm pretty sure they don't need to adjust much.

It's like two hours per epoch

But in one epoch it's gone from 0.5 to 0.6  balanced accuracy on the uncentered images, so it's relatively fast on a per epoch basis.

So this crashed overnight on the end of the second epoch for some reason...

No longer applicable. VGG was used instead


Can't build ASP.NET Core project using net461 and x64

All my projects are locked to x64 configuration (AnyCPU is deleted) because, well, it only runs on x64 and depends on x64 native binaries.
My ASP.NET core project is configured to use net461 as I need to the full .net framework in order to reference the Microsoft.WindowsAzure.ServiceBus nuget and to reference my class libraries.
project.json excerpt:
   "frameworks": {
    "net461": {}
  },

  "runtimes": {
    "win7-x64": {},
    "win8-x64": {},
    "win10-x64": {}
  },

  "buildOptions": {
    "emitEntryPoint": true,
    "preserveCompilationContext": true,
    "platform": "x64"
  },

Building the project now gives me an error:
2>C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets(724,5): error : The OutputPath property is not set for project 'BlackCat.Automation.Messaging.csproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='AnyCPU'.  This error may also appear if some other project is trying to follow a project-to-project reference to this project, this project has been unloaded or is not included in the solution, and the referencing project does not build using the same or an equivalent Configuration or Platform.
2>C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets(724,5): error : The OutputPath property is not set for project 'BlackCat.Core.csproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='AnyCPU'.  This error may also appear if some other project is trying to follow a project-to-project reference to this project, this project has been unloaded or is not included in the solution, and the referencing project does not build using the same or an equivalent Configuration or Platform.

I've tried manually changing the Any CPU => x64 in the sln file but it reverts back once you build the project.
Any ideas?
EDIT: Even after adding the AnyCPU configuration for the two projects referenced here, the project built successfully but while I tried to publish with win7-x64 runtime, it built the win10-x64 and couldn't find the output files. Switching to win10-x64 failed to start the app on Windows Server 2012 R2.
It seems to me that the whole .NET core ecosystem is very messy even at RC2 when it comes to supporting anything "non-standard" (limited to .NET Core apps). I would love to see an example project with this configuration: .NET Core APIs, to get advantage of all the goodies such as webpack etc but with reference to net461 - not all of us are interested in deploying to Linux right now...

Any updates? Issue still around, VS "fixes" the sln file and hardcodes Any CPU and then fails to build asp.net core project complaining one of its dependencies does not define Any CPU (of course it does not, it cannot in my case as the dependency is Service Fabric stateless service). Seriously frustrating experience considering such stateless services are officially recommended approach for out-of-cluster communication solution

Matt compiled a repro here
https://github.com/mattridgway/Service-Fabric-Error

I'm running into this issue as well. I'm using a shared PCL and need my ASP.NET Core service to be x64 & 4.6.1 for Azure Service Bus communication. If you set it to x64 in the configuration manager, it is automatically set back to Any CPU.

+1

We REALLY need a fix for this

I've managed to get a version working by using VS2017, I'm going to update the repo https://github.com/mattridgway/Service-Fabric-Error to show it working. It looks like the VS2017 tooling has fixed this, although your class libraries holding the interfaces MUST target the full .NET framework.

I ran into the same issue, with vs 2015 sp3, with full .net 4.6 (don't support Azure though, but has anyone found a workaround? - or some fix from ms?

+1. VS 2017. Create a new asp net core with .NET full framework. Project properties - change from x86 to x64. Web deploy publish to IIS. 502.5 error. Would be nice to work under x64! :)

Why can we not run under x64 or Any CPU?

+1 New Project in VS2017. Change to x64. Debug - BadImageFormatException.

I have problem with net452 and x64 as well. AnyCPU was fine, x64 causes problem.

A workaround is to 'Unload a Project' and edit runtime version in your .csproj file manually.
  <PropertyGroup>
    <TargetFramework>net47</TargetFramework>
    <RuntimeIdentifier>win10-x64</RuntimeIdentifier>
  </PropertyGroup>

It is very straightforward and works like a charm.

The problem is back with core 2.0... Setting RuntimeIdentifier does not help anymore...

This issue is being closed because it has not been updated in 3 months.
We apologize if this causes any inconvenience. We ask that if you are still encountering this issue, please log a new issue with updated information and we will investigate.


Updated viewport.getConstrainedBounds so it takes in current

…instead of immediately
FYI @josmilmed, this is what I was talking about in my last comment on your patch.

Ok, looks better.

Great; thanks for taking a look.


uv add pyqt5  notwork

error: Distribution pyqt5-qt5==5.15.15 @ registry+https://pypi.org/simple can't be installed because it doesn't have a source distribution or wheel for the current platform

This is a case of https://github.com/astral-sh/uv/issues/9711


[PE-D][Tester C] Unable to unlink csd(cabin service director) from flight even though it is present. Continuation of previous bug.

Step 1) first link a csd crew with slf command (previous bug) to  a flight
step 2) try to unlink the csd with a csd command

Labels: type.FunctionalityBug severity.Medium
original: Saha9616/ped#8

The fix for this is to rename the ranks to not clash with the flightpositions for crew members AND
also to display flightposition instead of rank for the crew in the flight panel since the link/unlink command rely on flightposition and not rank

Fixed with #235


Create a walkthrough for explorer steps

There are walkthroughs for meadow, garden and grapher steps, but not for explorer.
It should be easy to create cookiecutter templates for a simple explorer step, and the walkthrough script.

@paarriagadap once you make your Notion page "Creating Data Explorers" public, could you paste the link here? It's gonna be really useful to link to your doc in walkthrough explorers.


chore(main): release 4.1.1

:robot: I have created a release beep boop
4.1.1 (2024-07-30)
Bug Fixes

grepping or replacing in cwd instead of the directory of the file (#293) (aee19fb)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v4.1.1
:sunflower:


Moment should have operation_at

There is a method called this on Circuit. It would make sense to live on Moment and then the circuit method could dispatch to it

This sounds like a reasonable and simple improvement, though moment can be indexed with the qubit (moment[q] returns the operation on that qubit), isn't it superfluous to introduce operation_at?

Oh I didn't realize you could index into it like that. In any event, Circuit.operation_at should use that functionality

That is true, we could use moment[q] in Circuit.operation_at - just to clarify do you think it would be valuable to add operation_at to Moment as well as it might be more discoverable than the indexing?

I can't speak for everyone, but I would have found it if it was called operation_at :)

Hi, would it be possible for me to try and add operation_at to Moment? I might need to read the relevant codes first as I am not entirely familiar with them.

You got it! Thanks for volunteering!

ping @exAClior - are you looking at this? Thank you!

ping @exAClior - are you looking at this? Thank you!

Yes, I finished adding the feature, but I need to add tests for that piece of code. I will update it tonight.


Add option to set UDP TTL

Discussed in https://github.com/aler9/rtsp-simple-server/discussions/847

Originally posted by ThibautDur February 18, 2022
Hello !
I am currently using rtsp simple server to stream a video through a router.
But because the ttl of the packets is set to 1, the packages are discarded by the router.
Is there please a way to increase ttl for udp packets using rtsp protocol on rtsp simple server side (configuration option, flag,...) ?
Thank you for your answers !

golang.org/x/net/ipv4
ipv4.NewConn(udpConn).SetTTL(128)

Need to check how this is handled in GStreamer / FFmpeg.

golang.org/x/net/ipv4
ipv4.NewConn(udpConn).SetTTL(128)

Need to check how this is handled in GStreamer / FFmpeg.

FWIW I believe gstreamer defaults to 64 for regular UDP and 1 for multicast:
  ttl                 : Used for setting the unicast TTL parameter
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 64
  ttl-mc              : Used for setting the multicast TTL parameter
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 1

but there are some other cases like 16 here in gst-rtsp-server:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-rtsp-server/gst/rtsp-server/rtsp-sdp.c#L263

After some testing, i discovered that

unicast TTL is already 64 or 128 depending on the OS
multicast TTL is 1

Therefore, we need to adjust multicast TTL to a greater value.

added in v0.18.0


Ui position and performance bug

I made sure to have downloaded the right release before submitting the issue.

My mouse is on the message, but it thinks it's around where the ? is.

The first problem is low fps without any pathline.
The second problem is the wrong position of tooltip.
The third problem is wtf tap button is not clickable?? @LH-bored

just remove that tooltip and display its player name and pos. It would be much better and may solve low fps too

All I know is that there is a bug that can't be clicked, which happened on the mobile phone.
The other two you may want to ask @LtMZ1 .
@Sharlottes

All I know is that there is a bug that can't be clicked, which happened on the mobile phone.
The other two you may need to ask LtMZ1

I want to die because i cannot find the reason

Maybe you can try moving the UI on the left to the top of the screen, or try rolling back the UI part to before <IP_ADDRESS>?

Maybe you can try moving the UI on the left to the top of the screen, or try rolling back the UI part to before <IP_ADDRESS>?

why don;t you just use ui scaling?

UI scaling does not solve the clickable problem :(
@Sharlottes

UI scaling does not solve the clickable problem :(
@Sharlottes

did you update btw? It might be solved

UI scaling does not solve the clickable problem :(
@Sharlottes

did you update btw? It might be solved

I tried to update the version, but the MOD is still abnormal

I tried to update the version again, but the mod still has exceptions @Sharlottes

You can try marginTop(), marginLeft() to control the UI position
Of course, this may not be right

i will make instant setting ui, which you can config some unitinfo settings on ingame screen.
this includes those pathline buttons, which lead us to solution.

@Br0ad can you try it?


Limit PyTorch version

optimum-cli export openvino -m tiiuae/falcon-7b --weight-format int4 falcon-7b-ov-int4 results in an error with PyTorch 2.4 on Windows: The specified module could not be found. Error loading "[...]fbgem.dll" or one of its dependencies.
IPEX installs a lower version of PyTorch already so this change should be safe while we look into this.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@helena-intel  I suppose issue not into specific pytorch version, I found this thread related to your reported issue  https://discuss.pytorch.org/t/failed-to-import-pytorch-fbgemm-dll-or-one-of-its-dependencies-is-missing/201969/2 and it was reported in may, when torch 2.3 released. I suppose the recommendation for fixing that should be not limit pytorch version, but update VS C++ package

@helena-intel I suppose issue notiwith specific pytorch version, I found this thread related to your reported issue https://discuss.pytorch.org/t/failed-to-import-pytorch-fbgemm-dll-or-one-of-its-dependencies-is-missing/201969/2 and it was reported in may, when torch 2.3 released. I suppose the recommendation for fixing that should be not limit pytorch version, but update VS C++ package
That is annoying. Many of our Windows customers will run into this. But for me the issue was fixed after installing PyTorch 2.3, otherwise I would not have made the PR. I see that most of the comments from that issue are also from after the PyTorch 2.4 release so it seems possible there's a relationship.


Pipeline parsing of "pipeline.yml" failed (Expected identifier to start with a letter, got ?)

Similar to #584 I'm getting the error in the title trying to compare $? in a pipeline command step. Neither of these work regardless of quoting:
if [[ "\$?" == "0" ]]; then ...
if [[ "$$?" == "0" ]]; then ...

Hi @alanbccoe,
Are you open to emailing us a copy of the pipeline.yml and/or a link to a failing build? That will help us recreate the issue and understand what's going on. We're at<EMAIL_ADDRESS>

OK let me see if I can do that. I get the same issue trying to test a regex in a command block too:
if [[ ! "$i" =~ org/(unit1|unit2)/[^/]+/[^/]+$ ]]
if [[ ! "$i" =~ org/(unit1|unit2)/[^/]+/[^/]+\$ ]]

This should reproduce the issue (but I don't have time to test it right now unfortunately):
steps:
  - label: Validation
    command: |
      [[ "org/unit1/some/thing/else/invalid" =~ org/(unit1|unit2)/[^/]+/[^/]+\$ ]] && echo 'OK'

With evaluating exit codes on the command block, I find that the following works:
if [[ "$$\?" == "0" ]]; then ...


pip install requirements issue

Hello,
I wanted to give your software a try, but when I executed
pip install -r requirements.txt
I got this response.
I'm not very advanced and I'm hesitating if I should continue.
I imagine this message could scare away others as well.
What should I do to proceed ?
root@omv:/DATA/git/immich_duplicate_finder# pip install -r requirements.txt
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

i am just trying it ilke this
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
I followed the guide here
https://python.land/virtual-environments/virtualenv#How_to_create_a_Python_venv

Indeed it worked, but with this -pip install downloaded additional 5GB of data. A lot I must say...

This is to be expcted, dependencies on this project are huge. The generated docker image for this is above 6GB 🙄
Since you found a solution, please close this issue.


Bellman Equation in Latex

I didn't merge this pull request for a couple of reasons.

The excel file created problems because GitHub doesn't like excel files. So I'd rather wait until Taylor is done with that file and deal with it just one time.
I wanted to cheat my way around those extra commits we don't need.
Your repo doesn't match up to mine. It says its about 11 commits behind. We need to figure out why that is and get you synced. You don't have any of the folder structure that I put in place back at the beginning of the semester. It's like your fork hasn't been updating from my repo regularly. I'm going to email everyone about this in a minute.


Errors occur if the whole list of categories is not specified

Tested your extension and found something that could be improved,
The extension allows to work with specific categories which is very useful but if I don't specify a list of all the categories I have I get this kind of errors:
INFO    AccessOwnership: Movies Category: Invalid CategoryExt Name: ['Movies'] Name not same as in CATEGORIES Check inst        ructions in the extension settings
INFO    AccessOwnership: Series Category: Invalid CategoryExt Name: ['Series'] Name not same as in CATEGORIES Check inst        ructions in the extension settings
INFO    AccessOwnership: Software Category: Invalid CategoryExt Name: ['Software'] Name not same as in CATEGORIES Check
        instructions in the extension settings
ERROR   Script AccessOwnership with command Test failed

Expected behavior:
The extension should only check the categories I specified in its settings or if it is supposed to be like this, I would like to see some warning that I should specify the whole list of categories.

Yes, I am aware of that bug and the intended function is to only match the added categories that you specify in the extension. It has been fixed in next release. I have another version coming up any day now with added support for MacOS and some added functionality (detect UID/GID) and that bug will have been adressed.

Version 1.5 released with issue solved.


[RFE] Manila: add appendShareMetadata option

There's been a request for adding user-defined metadata to new shares. This ticket will track the respective PRs to Manila external provisioner as well as csi-manila.
The plan is to add a new field appendShareMetadata to StorageClass/volume parameters. This field will hold a JSON string containing KV-pairs, which will be then appended to the pre-defined metadata that's passed to Manila when creating a new share.
I'm working on this.
/kind feature

Done in #782


Bugfix/detect stations with 4 character

There exist currently 330 stations which have an id that
consist of only 4 characters. These generate a report link
of the form id_-BEOB.csv.
The regex for the href attribute was fixed such that trailing
underscores do not become part of the id.
The URL to retrieve the weather measurements was changed accordingly.

Thank you @mpg07 for your contribution! :)


go.*,vendor: Bump the k8s dependencies to v1.20.6

The v1.20.6 contains the github.com/gogo/protobuf version bump:
https://github.com/kubernetes/kubernetes/blob/v1.20.6/go.mod#L53.
Fixes CVE-2021-3121.

Description of the change:
Motivation for the change:
Reviewer Checklist

[ ] Implementation matches the proposed design, or proposal is updated to match implementation
[ ] Sufficient unit test coverage
[ ] Sufficient end-to-end test coverage
[ ] Docs updated or added to /doc
[ ] Commit messages sensible and descriptive

/cherry-pick release-4.7

/retest

/approve

/lgtm


netlink: bump to fix parsing of IFLA_IPTUN_COLLECT_METADATA

run tests

run functional-tests-backend-elasticsearch
run functional-tests-backend-orientdb

Upstream PR : https://github.com/vishvananda/netlink/pull/530


Memory leak?

This is my script for slide show:
#!/bin/sh
#<<<<<<<<< format: sh wallpapers.sh $TIMEOUT $DISPLAY_1 $DISPLAY_2
while (true)
do
    sleep 4
    LOOP_COUNT=1

    ls -1 ~/Obrazy/Wallpapers | grep -v webp  | grep -v mp4 | while read each
do
    if [ $(expr $LOOP_COUNT % 2 ) != "0" ]
    then
        hyprctl hyprpaper preload "~/Obrazy/Wallpapers/$each"
        sleep 1
        hyprctl hyprpaper wallpaper "$2,~/Obrazy/Wallpapers/$each"
    else
        hyprctl hyprpaper preload "~/Obrazy/Wallpapers/$each"
        sleep 1
        hyprctl hyprpaper wallpaper "$3,~/Obrazy/Wallpapers/$each"
        sleep $1 #<<<<<<<<<<<<<<<<<<<<<<<
        hyprctl hyprpaper unload all #<<<<<<<<<<<<<<<<<<<<<<
    fi

    LOOP_COUNT=$((LOOP_COUNT+1))
done
done

But "unload all" seems not to be working. hyprpaper grows in RAM with every loop iteration. Typing manually command for unload throws "ok", but doesn't work either.

doesn't seem tho be the case for me...
Checked with valgrind too, doesn't seem to be any leaks. There was one small leak of a few kB, fixed in 3bfaac09f58ce31c33e1a56e7eaa606a87fe4c32
apart from that, can't see anything.

What are you using for finding those leaks?

Checked with valgrind too

==2620679== LEAK SUMMARY:
==2620679==    definitely lost: 2,040 bytes in 15 blocks
==2620679==    indirectly lost: 1,384,455 bytes in 29,205 blocks
==2620679==      possibly lost: 185,662 bytes in 3,901 blocks
==2620679==    still reachable: 106,703,943 bytes in 10,120 blocks
==2620679==         suppressed: 0 bytes in 0 blocks
==2620679== Reachable blocks (those to which a pointer was found) are not shown.
==2620679== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2620679==
==2620679== For lists of detected and suppressed errors, rerun with: -s
==2620679== ERROR SUMMARY: 118 errors from 30 contexts (suppressed: 0 from 0)

Then it's not a leak but just unload is not doing it's job ¯_(ツ)_/¯

it is for me, odd.

This is my script for slide show:
#!/bin/sh
#<<<<<<<<< format: sh wallpapers.sh $TIMEOUT $DISPLAY_1 $DISPLAY_2
while (true)
do
    sleep 4
    LOOP_COUNT=1

    ls -1 ~/Obrazy/Wallpapers | grep -v webp  | grep -v mp4 | while read each
do
    if [ $(expr $LOOP_COUNT % 2 ) != "0" ]
    then
        hyprctl hyprpaper preload "~/Obrazy/Wallpapers/$each"
        sleep 1
        hyprctl hyprpaper wallpaper "$2,~/Obrazy/Wallpapers/$each"
    else
        hyprctl hyprpaper preload "~/Obrazy/Wallpapers/$each"
        sleep 1
        hyprctl hyprpaper wallpaper "$3,~/Obrazy/Wallpapers/$each"
        sleep $1 #<<<<<<<<<<<<<<<<<<<<<<<
        hyprctl hyprpaper unload all #<<<<<<<<<<<<<<<<<<<<<<
    fi

    LOOP_COUNT=$((LOOP_COUNT+1))
done
done

But "unload all" seems not to be working. hyprpaper grows in RAM with every loop iteration. Typing manually command for unload throws "ok", but doesn't work either.

I too see that the option unload all sometimes doesn't unload properly but does respond with ok .
I can start hyprpaper and i see it takes 7M at startup from btop. If I load my default image it rises to 26M but if I change to a larger image it rises to say 61M but when I switch back to my original image and run unload all it now reports 34M instead of the 26M .
All my wallpaper setting script is doing for my two screens is :
#!/bin/sh

# if pgrep swaybg; then
# 	pkill swaybg
# fi
# swaybg -m fill -i ~/.cache/wallpaper
read -r wallpaper<~/.cache/test
hyprctl hyprpaper preload "$wallpaper"
hyprctl hyprpaper wallpaper ", $wallpaper"
hyprctl hyprpaper unload all

Right, I see it, though it only happens once to me. Once the memory usage rises after a dynamic call, further calls will not increase ram after unload all to me.

no clue where it comes from, maybe a bug how it's measured? no dangling stuff is remaining and valgrind + lsan don't complain

I revisited hyprpaper again. Used script from original post. Doesn't seems to be an issue anymore for me.

i have a similar issue but in a longer timespan, my script is a bit more complicated but is because i have more images and 2 screens but here is my script:
#! /usr/bin/fish

set -g imgs ~/Dev/bgs/**.{jpg,jpeg,png,webp}
if test $status -ne 0
    set -f st $status
    echo "Bad Glob"
    return $st
end
set -g img_count (count $imgs)

function load_wallpaper
    set -f preload_cmd hyprpaper preload
    set -f wallpaper_cmd hyprpaper wallpaper

    set -f rand_num (random 1 $img_count)
    set -f curr_img $imgs[$rand_num]
    echo $curr_img
    hyprctl $preload_cmd $curr_img
    hyprctl $wallpaper_cmd "DP-1, $curr_img"
    sleep 2 && hyprctl $wallpaper_cmd "HDMI-A-1, $curr_img"
    hyprctl hyprpaper unload all
    dunstify -t 3000 "Wallpaper changed" "to $curr_img"
end

while true
    sleep $(math "60 * 5")

    load_wallpaper
end

it also happens in my laptop with the same script minus the sleep 2 line and with 1 screen.
Here is the last few lines of the log because i don't know for how much time it ran:
[LOG] Total SM usage for all buffers: 4.2MB
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Closing Accepted Connection
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Received a request: wallpaper DP-1, /home/sortextg/Dev/bgs/Wallpaper-Bank/Anime-Girl3.png
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Closing Accepted Connection
[LOG] configure for DP-1
[LOG] handlePreferredScale: 1.00 for 7739a80016c0
[LOG] handlePreferredScale: 1.00 for 7739a80016c0
[LOG] Image data for DP-1: /home/sortextg/Dev/bgs/Wallpaper-Bank/Anime-Girl3.png at [0.00, -11.27], scale: 0.70 (original image size: [1952, 1120])
[LOG] Rendering splash: "If it ain't broke, don't fix it" - Lucascito_03
[LOG] Splash color: 55ffffff
[LOG] Splash font size: 14, pos: 767.71, 1061.22
[LOG] Submitting viewport dest size 1366x768 for a80016c0
[LOG] configure for DP-1
[LOG] handlePreferredScale: 1.00 for 7739a80016c0
[LOG] handlePreferredScale: 1.00 for 7739a80016c0
[LOG] Image data for DP-1: /home/sortextg/Dev/bgs/Wallpaper-Bank/Anime-Girl3.png at [0.00, -11.27], scale: 0.70 (original image size: [1952, 1120])
[LOG] Rendering splash: "If it ain't broke, don't fix it" - Lucascito_03
[LOG] Splash color: 55ffffff
[LOG] Splash font size: 14, pos: 976.00, 1075.51
[LOG] Submitting viewport dest size 1366x768 for a80016c0
[LOG] handlePreferredScale: 1.00 for 7739a80016c0
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Received a request: wallpaper HDMI-A-1, /home/sortextg/Dev/bgs/Wallpaper-Bank/Anime-Girl3.png
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Closing Accepted Connection
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Received a request: unload all
[LOG] Unloading target /home/sortextg/Dev/bgs/Wallpaper-Bank/Anime-Girl3.png, preload path /run/user/1000/.hyprpaper_GwUbiH
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Closing Accepted Connection
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Received a request: preload /home/sortextg/Dev/bgs/Wallpaper-Bank/Lofi-Desktop-Man-Studying.png
[LOG] Preloaded target /home/sortextg/Dev/bgs/Wallpaper-Bank/Lofi-Desktop-Man-Studying.png in 143.94ms -> Pixel size: [2088, 1080]
[LOG] Buffer created for target /home/sortextg/Dev/bgs/Wallpaper-Bank/Lofi-Desktop-Man-Studying.png, Shared Memory usage: 4.2MB
[LOG] Total SM usage for all buffers: 4.2MB
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Closing Accepted Connection
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Received a request: wallpaper DP-1, /home/sortextg/Dev/bgs/Wallpaper-Bank/Lofi-Desktop-Man-Studying.png
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Closing Accepted Connection
[LOG] configure for DP-1
[LOG] handlePreferredScale: 1.00 for 7739a8005350
[LOG] handlePreferredScale: 1.00 for 7739a8005350
[LOG] Image data for DP-1: /home/sortextg/Dev/bgs/Wallpaper-Bank/Lofi-Desktop-Man-Studying.png at [-83.53, 0.00], scale: 0.71 (original image size: [2088, 1080])
[LOG] Rendering splash: "If it ain't broke, don't fix it" - Lucascito_03
[LOG] Splash color: 55ffffff
[LOG] Splash font size: 14, pos: 756.78, 1044.34
[LOG] Submitting viewport dest size 1366x768 for a8005350
[LOG] configure for DP-1
[LOG] handlePreferredScale: 1.00 for 7739a8005350
[LOG] handlePreferredScale: 1.00 for 7739a8005350
[LOG] Image data for DP-1: /home/sortextg/Dev/bgs/Wallpaper-Bank/Lofi-Desktop-Man-Studying.png at [-83.53, 0.00], scale: 0.71 (original image size: [2088, 1080])
[LOG] Rendering splash: "If it ain't broke, don't fix it" - Lucascito_03
[LOG] Splash color: 55ffffff
[LOG] Splash font size: 14, pos: 960.47, 1058.40
[LOG] Submitting viewport dest size 1366x768 for a8005350
[LOG] handlePreferredScale: 1.00 for 7739a8005350
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Received a request: wallpaper HDMI-A-1, /home/sortextg/Dev/bgs/Wallpaper-Bank/Lofi-Desktop-Man-Studying.png
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Closing Accepted Connection
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Received a request: unload all
[LOG] Unloading target /home/sortextg/Dev/bgs/Wallpaper-Bank/Lofi-Desktop-Man-Studying.png, preload path /run/user/1000/.hyprpaper_GqKFmi
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Closing Accepted Connection
[LOG] Accepted incoming socket connection request on fd 4
[LOG] Received a request: preload /home/sortextg/Dev/bgs/Wallpaper-Bank/Anime-Girl4.png
[CRITICAL] Failed to read image /home/sortextg/Dev/bgs/Wallpaper-Bank/Anime-Girl4.png because of:
out of memory

Did'nt test with valgrind btw, maybe another time with a simpler script


BasicModal的defaultFullscreen可以进行监听然后重新计算表格高度吗

可以监听modal的那个全屏按钮事件吗？

现实很骨感，提了个 pr 实现 https://github.com/vbenjs/vue-vben-admin/pull/3549/files
@zj1312114390

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days


Fix waiting order of electrons in docs

The document says for task A to wait on task B but the snippet was doing the opposite.

[ ] I have added the tests to cover my changes.
[ ] I have updated the documentation and CHANGELOG accordingly.
[x] I have read the CONTRIBUTING document.

Thanks @madhur-tandon ! great eye, we missed this one !!


ODI Docker Version

Hi
I'm using the odi docker image for setting up odi agents for multiple databases. Now i'm using the odi version <IP_ADDRESS>.0, but in future i need a newer version of odi. Is it planned, that future versions of odi (like <IP_ADDRESS>.0) will be added to this repository?

Found new version (https://www.oracle.com/middleware/technologies/data-integrator-downloads.html)


fix: remote unix machine x window issue when running specific sas cod…

Cannot run PROC IMPORT on remote Unix machine (#699)
Summary
When using the SSH to login the Unix machine, we supplied a wrong SAS options to connect the session.
The "-terminal" should be changed to "-noterminal".
(https://go.documentation.sas.com/doc/en/pgmsascdc/v_045/lesysoptsref/p1db6atd7mw5fnn1lvcra2s1i7tq.htm)
Testing

Add a new connection profile which support the SAS9.4 (remote -SSH);
Try to run the line
proc import datafile=x out=_csv dbms=csv replace; run;
The output should not be:
ERROR: The connection to the X display server could not be made. Verify that
the X display name is correct, and that you have access authorization.
See the online Help for more information about connecting to an X
display server.
ERROR: Device does not support full-screen.

It should be: ERROR: Physical file does not exist.....

adding @smorrisj who originally authored it for review

We have no proper environment to reproduce issue (https://github.com/sassoftware/vscode-sas-extension/issues/699).  Approve the fix and the consumer can verify it after the release.


Bug 1768417: Adjust UsingDeprecatedAPIExtensionsV1Beta1 alert for kube changes

/hold
We can't be sure the calert works and need to check manually because this alert is ignored - we can't unignore it at the same time because those are different repos.
/cc @soltysh

/bugzilla refresh

/retest


DOC: scipy.ndimage.convolve origin parameter

Issue with current documentation:
https://scipy.github.io/devdocs/reference/generated/scipy.ndimage.convolve.html#scipy-ndimage-convolve
The ndimage.convolve has the parameter origin, which is listed as type int.
when working with this, it actually appears to take in a sequence with length equal to the number of dimensions.
Idea or request for content:
clarifying the description to be the same as ndimage.correlate:
https://scipy.github.io/devdocs/reference/generated/scipy.ndimage.correlate.html#scipy.ndimage.correlate
Additional context (e.g. screenshots, GIFs)
No response

@AlexanderForrest would you like to make a PR

I also ran into this discrepancy - the documentation not listing axis-wise origin (offset) values was very surprising as well.

if you could add any suggestions over at gh-20408, that would be great @milindsmart


Performance for purely unsupervised training

Hi, I notice in the paper the unsupervised part is starting from a pretrained bert model, however, would it achieve similar results if train entirely from scratch, Thank you

Hi,
Yes unsupervised simcse needs to be trained from a pre-trained checkpoint. Training from scratch won't work.


Remove trailing whitespace from make-release.sh

What does this PR do?
Trim trailing whitespace from make-release.sh, which I noticed while rebasing https://github.com/devfile/devworkspace-operator/pull/296

/test v5-devworkspaces-operator-e2e


feat: mysql integration (DO NOT MERGE)

Summary of changes:

Updated bundle to use mysql-k8s charm.

Needed to do signed commit. Closing this PR.


Packagist versions

Hello,
Is there a reason that v2.0.1 and v2.0.2 are not published in packagist?
When looking at the published packages I just see 2.0.
I'd like to use your package in my project but using dev-master is not an option for me.
Thanks a lot for your response,
he8us

Hi @he8us here you are.
https://packagist.org/packages/bulutyazilim/feedback-bundle#2.0.3

Hey thank you


[RangeError] use missing size parameter

Using this serde declaration:
sd.use(sd.array(sd.bytes(sd.uint32), sd.uint32));

I get the following error:
RangeError: offset is out of bounds
    at Uint8Array.set (<anonymous>)
    at Object.ser (.../node_modules/sirdez/src/serdes/bytes.ts:9:17)
    at .../node_modules/sirdez/src/serdes/array.ts:10:12
    at contextSer (.../node_modules/sirdez/src/context.ts:27:7)
    at Object.toBytes (.../node_modules/sirdez/src/use.ts:14:34)
    at serialize (.../src/datasources/srs.ts:391:42)
    at CacheClient.set (.../src/lib/redis-cache/cache.js:203:48)
    at CacheClient.tryGet (.../src/lib/redis-cache/cache.js:131:24)

Looking at the source code I should be able to create a context with a larger size than the default value. For that I would expect to call use second parameter, size with a size.
However, the signature available for sd.use in my current project is this:
function use<T>(
  { ser, des }: Serdes<T>
): UsableSerdes<T>

I was expecting to have something like this:
function use<T>(
  { ser, des }: Serdes<T>,
  size?: number
): UsableSerdes<T>

package version: 0.0.6

What am I doing wrong?

Yes, the package is outdated, I haven't published to NPM yet. I need to make sure everything is right before the push. Although, it is strange that you are getting this error. The array buffer should grow by itself upon encountering this error. I will investigate this error tomorrow if I have time and publish it.

I was able to reproduce this bug with the following code:
import * as sd from 'sirdez';

const contextByteLength = 4096;
const headByteLength = 2;

const data = new Uint8Array(contextByteLength - headByteLength + 1);
const serdes = sd.use(sd.bytes(sd.uint16));

serdes.toBytes(data);

This code throws exception RangeError: offset is out of bounds. The exception happens here because data array is too big to fit into context.
The context size is the default one used in code - 4096 bytes, 2 bytes are taken for head, so we have only 4094 bytes left. The data array is one byte bigger, so it doesn't fit, and Uint8Array.set(data, offset) throws error with misleading message offset is out of bounds. Actually, the offset is in bounds, it just that data doesn't fit the buffer.
Next, the exception is caught here. When serializing data, there might happen 2 types of errors: error caused by small size of the buffer, and some user error unrelated to the buffer. If error is caused by the buffer size, we need to ignore it and grow the buffer. If error is a user error, we need to re-throw it.
The code assumes that if ctx.i (i.e. the last offset) is smaller than the limit of the current buffer, then error is the user error. In this case this is not true, because, as I've written earlier, the last offset is actually in buffer bounds: the error happens because the data doesn't fit fully into buffer.
So, maybe we should replace if (ctx.i < limit) throw error; on that line with
if (!(error instanceof RangeError)) throw error;

That is, assume that every RangeError is caused by the buffer size and ignore it?

Another solution that will make use of the if guard if (ctx.i < limit) throw error is to increment the pointer i before Uint8Array.set(data, offset).
I think adding both the limit check and the instanceof check will further narrow the errors that are actually caused by out-of-bounds error.

I hope this issue only concerns bytes.ser.

I made some changes and some tests reflecting the suggestion above, hopefully, it is fine. https://github.com/weisrc/sirdez/issues/11#issuecomment-1295448084

It seems like in Uint8Array, only set() method can throw error if input data is too large, and my code search says that set() is used only in bytes.ser, so the solution should be fine. Though it's easy to forget about this trick if someday some other code will use set() again.
Maybe add warning for Uint8Array.set() with this ESLint rule? Something like "Context pointer must be incremented before calling Uint8Array.set(), see issue #11". I can add PR tomorrow


Path invalid in Windows for scapegoat-scalastyle.xml

scapegoat-scalastyle.txt
In the report, all slashes are replaced by dots. So I can not get the reference of the file in my continuous integration server
Example :
C:.Program Files.Jenkins.jobs.backofficeapi.RouteService.scala
instead of
C:\Program Files\Jenkins.jobs\backofficeapi\RouteService.scala
Can you fix this please ?
Thank you

Hello
I can confirm and I have also this issue on MacOS. scapegot.html, scapegot.xml and scapegot-scalastyle.xml are generated with dotes instead of slashes.
When I launch sbt-sonar, the result of scapegot-scalastyle.xml is ignored.

Fail to resolve 100 file path(s) in Scapegoat report. No issues imported related to file(s):


refactor processing clients packets

postgresql has 3 special message types that has format another than all other message: startup, ssl request and cancelation request.
before we try to parse only first packet as special and all other as general
but was case when client send ssl request (first special), then if it continue after db response it send startup message. we correctly handle case when db accept ssl request and didn't handle deny from db.
Now processing re-writed for client's side of processing and we try to recognize these special messages or other known general messages and doesn't depend on their order (first, second etc). Plus in a future can process them separately.
plus added handling columns with null values (it fixes panics [T790])

Cool that you've added unit tests to catch this!


refactor: update Imperva logo

Update Imperva logo svg.
The current logo used for Imperva is their white text version, and it's invisible in the web extension - I've changed the logo to a coloured version, the same used as their website favicon.

Thanks!


support docker-compose.yaml, fix #44

@surajssd @kadel, updated the PR.

In libcompose docker-compose.yml is default https://github.com/docker/libcompose/blob/master/cli/command/command.go#L341

LGTM

Added a couple of simple tests for this, added the comment. Will squash once you guys review, and LGTM!

:+1:  LGTM

great work @containscafeine


Rename M2351 platform name to 'NU_PFM_M2351' from 'NUMAKER_PFM_M2351'

Description
Originally its name is NUMAKER_PFM_M2351. After adding suffix to distinguish PSA/non-PSA targets, it will have pattern below which exceeds max 20 chars limited by online database:
'NUMAKER_PFM_M2351'+'_'+['PSA', 'NOPSA']+'_'+['S', 'NS']

To fix the issue, the M2351 platform name is changed to NU_PFM_M2351. Then it will have pattern below for PSA/non-PSA targets, which fit in max 20 chars limit:
'NU_PFM_M2351'+'_'+['P', 'NP']+'_'+['S', 'NS']

Related PR
https://github.com/ARMmbed/mbed-os/pull/11288
Pull request type
[ ] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[ ] Test update
[x] Breaking change

Reviewers
@MarceloSalazar @mark-edgeworth

@ccli8 Assuming the related PR gets in 5.14.0, this should be integrated with it (asap) ?

Assuming the related PR gets in 5.14.0, this should be integrated with it (asap) ?

Yes.

@mark-edgeworth I agree and I'll make the changes once we release 5.14.0, if that's ok?


change default connection name to hostname

When we create a new connection to the database server, the default connection name is database name
So, let's say in Oracle it is like SID it doesn't mean anything, doesn't do understanding on which server i connect. Or in postgresql i usually connect to postgres database.
Having many database server i connect on each server once, then I choose a specific database from top menu
Describe the solution you'd like
it would be more convenient to set default connection name to hostname of database server

@cloud-66 Thank you for the suggestion, we will look if it can be implemented in future updates. For now you can enable showing of host name in the connection's context menu.


Migrate to Python 3.6

I wanted to use this with my python 3 project, so I migrated it.
This is just a run of 2to3 -w on the project, and updating the readme.
I'm running it successfully locally on 3.6.

Amazing! thanks! merged!


Treasury - Operations > Payments: Payment Recipients Page

Note: This ticket has a dependency on https://github.com/NEAR-DevHub/neardevhub-treasury-dashboard/issues/18.
User Stories

As an admin with create permissions, I want to view, add, edit, and delete payment recipients so I can manage the list of approved recipients for outgoing payments.
As an admin with approve permissions, I want to view the list of payment recipients so I can understand who is eligible to receive payments.
Regular users cannot see payment recipients

Acceptance Criteria

The "Pending Request" page is accessible under Operations > Payments > Payments in the Treasury dashboard.
The sidebar displays the total number of recipients next to the "Payment Recipients" title.
The "Payment Recipients" right content section displays a list of all payment recipients, including the following columns:

ID: The ID of the recipient (default sort, ascending order from 01).
First Name: The first name of the recipient.
Last Name: The last name of the recipient.
Email (hidden but captured, can be viewed in edit)
Org: The organization the recipient belongs to (if applicable).
Wallet: The wallet address of the recipient.
KYC/B Verified: The KYC/KYB verification status of the recipient (Verified or not verified)
Test Transaction: The status of the test transaction sent to the recipient's wallet (Confirmed or Not Confirmed).
Transactions: The number of transactions made to this recipient.
Total Paid: The total amount paid to this recipient.
Actions: Buttons for editing (pencil icon) and deleting (trashcan icon) the recipient (visible only to admins with create permissions).

There is a "New Recipient" button that admins with create permission can click to create a new recipient
There is pagination at the bottom to handle a large number of requests (e.g. greater than 20).
Sorting options are available to organize the requests by clicking on the column header.
Filter options are available (e.g. viewing people from a certain organization)
Permissions

Regular users cannot see the Payment Recipients tab
Admins with approve permissions can only view Payment Recipients
Admins with create permissions can create, edit, and delete recipients

Attachments

It's completed


FLASC dependency in LookupBasedWakeSteeringController

As noted in #31 , there are currently dependency conflicts between FLASC and FLORIS. The only reason that FLASC is required in WHOC is for the LookupBasedWakeSteeringController.
Possible solutions:

Disable the LookupBasedWakeSteeringController and remove FLASC dependencies (#31 ); then reinstate them when FLASC has been brought up to speed with Floris v4
Implement a local solution for get_yaw_angles_interpolant, which is the only functionality imported from FLASC, and make the removal of the dependency on FLASC permanent.

This was addressed in #31 and #35


single line items not available in widescreen

The feed icon/name and item title take up multiple lines in widescreen mode (see screenshot), differing from the single line they use in screenshots for this project. In not-widescreen mode, the single line name/title seems to work fine.
The problem appears for me in both Chrome and Firefox. How should I help debug this problem?

Hi @benzipperer, this is intended since in widescreen mode the overview area can be very narrow. The headline should have as much space as possible there.
Also, in widescreen mode, the excerpt breaks into multiple lines, otherwide it's on the same line like the icons and actions.


AX PRO  not getting zone update message for second zone

hi! i manage to run node-red with this library, connect to AX Pro, and setup a simply mqtt broker to send status / change setteings via mqtt commands. What is interesting for me
when i am playing with the APP hik connect or directly from node-red i see
AXPRO {"zoneUpdate":{"id":0,"name":"Strefa przewodowa 1","status":"online","sensorStatus":"normal","tamperEvident":true,"shielded":false,"bypassed":false,"armed":true,"isArming":false,"alarm":false,"subSystemNo":1,"linkageSubSystem":[1],"detectorType":"passiveInfraredDetector","stayAway":false,"zoneType":"Instant","accessModuleType":"localTransmitter","moduleChannel":1,"zoneAttrib":"wired","deviceNo":2,"abnormalOrNot":false}}
when i send command to "whole alarm system" or to First zone (id number: 0). I have 2 zones in the setup and don't see any status updates for second zone even when i am playing with the zone from himkvision app.
but i do see the CIDevents for different zones i think system 1 or 2
"CIDEvent":{"code":3441,"name":"jkjebj","type":"armAndDisarm","trigger":"2023-03-13T13:59:00+01:00","upload":"2023-03-13T13:59:02+01:00","system":1,"subSystemName":"Dom parter"}}
AXPRO {"CIDEvent":{"code":1401,"name":"jkjebj","type":"armAndDisarm","trigger":"2023-03-13T14:00:30+01:00","upload":"2023-03-13T14:00:30+01:00","system":2,"subSystemName":"Dom pietro"}}
AXPRO {"CIDEvent":{"code":3401,"name":"jkjebj","type":"armAndDisarm","trigger":"2023-03-13T14:00:38+01:00","upload":"2023-03-13T14:00:38+01:00","system":2,"subSystemName":"Dom pietro"}}
AXPRO {"CIDEvent":{"code":3441,"name":"jkjebj","type":"armAndDisarm","trigger":"2023-03-13T14:00:39+01:00","upload":"2023-03-13T14:00:40+01:00","system":2,"subSystemName":"Dom pietro"}}
not sure if this is a bug we can fix, happy to help you with testing

Hi
the arming works only if you have something to arm in the zone.
The Hik APIs reports only armed devices bellnging to a zone.


Document custom runtime flavors

Issue Ref: #900
Description:
Document how to create custom runtimes based on official ones.
TODOs:

[X] Ready to review
- [ ] Automated Tests
[X] Docs

Hi @andresmgot，I recently started researching your project kubeless，and it's awesome!
I have a question, is it possible to custom define a C runtime?

@andresmgot  I just read the doc of implement a custom runtime How to implement a new Kubeless run time,

but here in the red box,the link goes to a 404,is it normal?

hi @laurencechan, it's indeed possible to create a runtime for C. I think you can take the existing Go one as an example since I think it's the most similar one.
Thanks for reporting the broken link! I will remove it (it should just point to the point 2.)

hi @andresmgot .Thanks for your feedback！I went through the Go runtime as you told me recently,and checked the golang Dockerfile.init.But I didn't find any words similar to "go build" to compile the the webserver file 'kubeless.tpl.go',and in the golang Dockerfile ,we can see the line 'CMD ["/kubeless/server"]',where did the execuable file server come from?
github.com\kubeless\kubeless\docker\runtime\golang\Dockerfile.init
`FROM golang:1.10
Install dep
RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
RUN go get golang.org/x/net/context
RUN mkdir -p /go/pkg/dep
Prepare function environment
RUN ln -s /kubeless $GOPATH/src/kubeless
Install controller
RUN mkdir -p $GOPATH/src/github.com/kubeless/kubeless/pkg/functions
ADD pkg/functions/* $GOPATH/src/github.com/kubeless/kubeless/pkg/functions
ADD pkg/function-proxy $GOPATH/src/github.com/kubeless/kubeless/pkg/function-proxy
WORKDIR $GOPATH/src/github.com/kubeless/kubeless/pkg/function-proxy
RUN dep ensure
RUN mkdir -p $GOPATH/src/controller
ADD docker/runtime/golang/Gopkg.toml $GOPATH/src/controller/
WORKDIR $GOPATH/src/controller/
ADD docker/runtime/golang/kubeless.tpl.go $GOPATH/src/controller/
RUN dep ensure
RUN chmod -R a+w $GOPATH
RUN chmod -R a+w /go/pkg/dep
USER 1000
\github.com\kubeless\kubeless\docker\runtime\golang\DockerfileFROM bitnami/minideb:jessie
RUN install_packages ca-certificates
USER 1000
CMD [ "/kubeless/server" ]
`

hi @andresmgot, I create a function by the command kubeless function deploy lolo --runtime go1.10 --handler helloget.Foo --from-file helloget.go successfully
[root@localhost golang]# kubeless function ls
hellohellohellohelloNAME        NAMESPACE       HANDLER                 RUNTIME         DEPENDENCIES    STATUS
lolo            default         helloget.Foo            go1.10                          1/1 READY

Then I check the function pod , I can see the compile command in command args,which is " sed 's/<>/Foo/g' $GOPATH/src/controller/kubeless.tpl.go > $GOPATH/src/controller/kubeless.go && go build -o /kubeless/server $GOPATH/src/controller/kubeless.go > /dev/termination-log 2>&1" But I have no idea where this command comes from ??
[root@localhost golang]# kubectl get pod
NAME                        READY     STATUS    RESTARTS   AGE
kel-5465b57b6d-6zkjt        1/1       Running   0          19d
lolo-699f68589f-k6ww7       1/1       Running   0          1h

[root@localhost golang]# kubectl describe pod lolo-699f68589f-k6ww7
Name:               lolo-699f68589f-k6ww7
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               localhost.localdomain/<IP_ADDRESS>
Start Time:         Tue, 11 Sep 2018 18:10:10 +0800
Labels:             created-by=kubeless
                    function=lolo
                    pod-template-hash=2559241459
Annotations:        cni.projectcalico.org/podIP=<IP_ADDRESS>/32
                    prometheus.io/path=/metrics
                    prometheus.io/port=8080
                    prometheus.io/scrape=true
Status:             Running
IP:                 <IP_ADDRESS>
Controlled By:      ReplicaSet/lolo-699f68589f
Init Containers:
  prepare:
    Container ID:  docker://a9c981a9ff781d6d0dfeb58473ec4471fc8f5e7810f8b055273bfb8421b06a4f
    Image:         kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697
    Image ID:      docker-pullable://kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      echo '0bb4abb71663527961cd8d5420800be5499fae754b26a37359c07e598c01d1f6  /src/helloget.go' > /tmp/func.sha256 && sha256sum -c /tmp/func.sha256 && cp /src/helloget.go /kubeless/helloget.go && cp /src/Gopkg.toml /kubeless
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 11 Sep 2018 18:10:11 +0800
      Finished:     Tue, 11 Sep 2018 18:10:11 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /kubeless from lolo (rw)
      /src from lolo-deps (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jznxq (ro)
  compile:
    Container ID:  docker://2bc3d8d8edef81112be054267c71cd44599a10212618c37489a074903efb7264
    Image:         kubeless/go-init@sha256:983b3f06452321a2299588966817e724d1a9c24be76cf1b12c14843efcdff502
    Image ID:      docker-pullable://kubeless/go-init@sha256:983b3f06452321a2299588966817e724d1a9c24be76cf1b12c14843efcdff502
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      sed 's/<<FUNCTION>>/Foo/g' $GOPATH/src/controller/kubeless.tpl.go > $GOPATH/src/controller/kubeless.go && go build -o /kubeless/server $GOPATH/src/controller/kubeless.go > /dev/termination-log 2>&1
    State:      Terminated
      Reason:   Completed
      Message:  go: disabling cache (/.cache/go-build) due to initialization failure: mkdir /.cache: permission denied

      Exit Code:    0
      Started:      Tue, 11 Sep 2018 18:11:48 +0800
      Finished:     Tue, 11 Sep 2018 18:11:54 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /kubeless from lolo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jznxq (ro)
Containers:
  lolo:
    Container ID:   docker://636acd5fc7a6fa686075e29454a55a2071b81db57b7fe56dc5c793df2051ba7d
    Image:          kubeless/go@sha256:e2fd49f09b6ff8c9bac6f1592b3119ea74237c47e2955a003983e08524cb3ae5
    Image ID:       docker-pullable://kubeless/go@sha256:e2fd49f09b6ff8c9bac6f1592b3119ea74237c47e2955a003983e08524cb3ae5
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 11 Sep 2018 18:12:12 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8080/healthz delay=3s timeout=1s period=30s #success=1 #failure=3
    Environment:
      FUNC_HANDLER:       Foo
      MOD_NAME:           helloget
      FUNC_TIMEOUT:       180
      FUNC_RUNTIME:       go1.10
      FUNC_MEMORY_LIMIT:  0
      FUNC_PORT:          8080
    Mounts:
      /kubeless from lolo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jznxq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  lolo:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  lolo-deps:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      lolo
    Optional:  false
  default-token-jznxq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jznxq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

hi @andresmgot finally i find the go build action in the function named GetCompilationContainer in file kubeless\kubeless\pkg\langruntime\langruntime.go.
But i still didn't figure out how did the code go into this file after i deployed a new function using golang runtime.
The function GetCompilationContainer was relevant with the function-controller.But  when I use "kubeless function deploy" to create a new function,how did the code go into the function-controller.go file.I mean, how can I use the cmd/function-controller/function-controller.go the same as i use the cmd/kubeless/kubeless.go? And what's the relationship between the cmd kubeless and function-controller?

Hi @laurencechan, that the cmd does is create a CRD with the information of the function (including the code). This K8s object is picked by the controller and it's the one that generates the pods and containers that run the compilation, installation of dependencies, etc. These docs may help you clarify things:
https://kubeless.io/docs/advanced-function-deployment/
http://kubeless.io/docs/architecture/

Hi @andresmgot Thanks for you reply!I almost understand what you are talking about.The cmd create a CRD with all the information about the function you offered,and leave the k8s to create function pod and service according the CRD.
But I still have a some puzzlements.
The kubeless cmd package is divided into two subpackages,function-controller and kubeless.The entrance method of deploying a new function through kubeless cmd would be the "main" function from file "/cmd/kubeless/kubeless.go",I use gdb to debug the code and trace the route of the code.At last, the programe goes into the function "NewRequest" in file "/go/src/net/http/request.go",the code shows as below:
func NewRequest(method, url string, body io.Reader) (*Request, error) {
        if method == "" {
                // We document that "" means "GET" for Request.Method, and people have
                // relied on that from NewRequest, so keep that working.
                // We still enforce validMethod for non-empty methods.
                method = "GET"
        }
        if !validMethod(method) {
                return nil, fmt.Errorf("net/http: invalid method %q", method)
        }
        u, err := parseURL(url) // Just url.Parse (url is shadowed for godoc).
        if err != nil {
                return nil, err
        }
        rc, ok := body.(io.ReadCloser)
        if !ok && body != nil {
                rc = ioutil.NopCloser(body)
        }
        // The host's colon:port should be normalized. See Issue 14836.
        u.Host = removeEmptyPort(u.Host)
        req := &Request{
                Method:     method,
                URL:        u,
                Proto:      "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header:     make(Header),
                Body:       rc,
                Host:       u.Host,
        }
        if body != nil {
                switch v := body.(type) {
                case *bytes.Buffer:
                        req.ContentLength = int64(v.Len())
                        buf := v.Bytes()
                        req.GetBody = func() (io.ReadCloser, error) {
                                r := bytes.NewReader(buf)
                                return ioutil.NopCloser(r), nil
                        }
                case *bytes.Reader:
                        req.ContentLength = int64(v.Len())
                        snapshot := *v
                        req.GetBody = func() (io.ReadCloser, error) {
                                r := snapshot
                                return ioutil.NopCloser(&r), nil
                        }
                case *strings.Reader:
                        req.ContentLength = int64(v.Len())
                        snapshot := *v
                        req.GetBody = func() (io.ReadCloser, error) {
                                r := snapshot
                                return ioutil.NopCloser(&r), nil
                        }
                default:
                }
                if req.GetBody != nil && req.ContentLength == 0 {
                        req.Body = NoBody
                        req.GetBody = func() (io.ReadCloser, error) { return NoBody, nil }
                }
        }

        return req, nil
}

I use "info args" to show the args,It shows as below:

From here, I got two questions:
1.which server will handle this POST method through the url https://<IP_ADDRESS>:6443/apis/kubeless.io/v1beta1/namespaces/default/functions?and what's the return value?
2.until now ,the code route has never access any code from the function-controller package,how did it use the code in function-controller to form the CRD it need?
Waiting for your feedback,Thank you so much~!

@laurencechan https://<IP_ADDRESS>:6443/apis/kubeless.io/v1beta1/namespaces/default/functions is reaching the K8s API server. The goal of accessing that REST API is to create a Function object (with a POST). As you say, at that point the function-controller has not done anything. When the object is created is when the function controller detects that there is a new Function object and with that information it generates all the other k8s resources needed.
This blogpost can help you understand the controller model: https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html

Hi~ @andresmgot Thanks a lot for your reply, I'll check this blog carefully,Thank you so much~~

Hi~ @andresmgot Sorry to bother you again,after read the blog you recommended and other materials about kubeless and k8s,I still have some puzzles.
First, the url https://<IP_ADDRESS>:6443/apis/kubeless.io/v1beta1/namespaces/default/functions must be a aggregated api extended by your team based on the k8s style.you said the goal of accessing that REST API is to create a Function object,is it just like we use "kubectl create -f xxx.yaml" to create a API object?and how can i specify the particular method corresponding to this very url here.
Second,the function controller detect the new Function,and this process is consistent with that described in this figure in this url.is that right?
Third,you said that the function controller will generates all the other k8s resources needed,is that include the init-container information,compilation information,etc.It actually fulfill the function object (just like to change the A_version.yaml to a B_version.yaml).
I felt a little confused and vague,Please give me some advice.Thank you so much.
If you have other interesting materials about this,maybe you can show me through my gmail, laurencechan88@gmail.com,Thank you so much!!!

Hi~ @andresmgot  I use only one-node cluster and i commit a custom runtime image.I tag it kubeless/mypython:2.0,showed as below:

I used the command below to deploy a function used the new runtime image

But it turns out to be a errimagepull

all the images I use are in the local storage, should I still have to set up a registry?
I tried to pull the image,but failed,and I have executed docker login and success.

It seems all this images are in the same registry, how come is it that I just can pull images with a version  successfully and failed with the other ones ?

Hi, kubeless/ is the official registry, you cannot push images to that repository. What you can do is change the registry to your personal docker hub account (something like laurenchan/mypython) or configure the function to use a local image (deploying it with the flag kubeless deploy function --image-pull-policy=IfNotPresent ...)

Hi~ @andresmgot Thank you so much, It works!
But i got another problem,I tried to add a c runtime, and I edit the function GetCompilationContainer in file /root/go/src/github.com/kubeless/kubeless/pkg/langruntime/langruntime.go,then I rebuild the file /root/go/src/github.com/kubeless/kubeless/cmd/kubeless/kubeless.go, but when I deploy a C function, I got an error in function-controller container as below:
ERROR: logging before flag.Parse: E0926 17:43:08.042857       1 function_controller.go:185] Not found a valid compilation step for c1.0
time="2018-09-26T17:55:01Z" level=info msg="Processing change to Function default/crtest" pkg=function-controller
time="2018-09-26T17:55:01Z" level=info msg="Function object default/crtest has been successfully processed and marked for deleteion" pkg=function-controller
time="2018-09-26T17:55:01Z" level=info msg="Processing change to Function default/crtest" pkg=function-controller
time="2018-09-26T17:55:01Z" level=info msg="Function object default/crtest not found in the cache, ignoring the deletion update" pkg=function-controller
time="2018-09-26T17:58:09Z" level=info msg="Processing change to Function default/crtest" pkg=function-controller
time="2018-09-26T17:58:09Z" level=error msg="Function can not be created/updated: Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:09Z" level=error msg="Error processing default/crtest (will retry): Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:09Z" level=info msg="Processing change to Function default/crtest" pkg=function-controller
time="2018-09-26T17:58:09Z" level=error msg="Function can not be created/updated: Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:09Z" level=error msg="Error processing default/crtest (will retry): Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:09Z" level=info msg="Processing change to Function default/crtest" pkg=function-controller
time="2018-09-26T17:58:10Z" level=error msg="Function can not be created/updated: Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:10Z" level=error msg="Error processing default/crtest (will retry): Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:10Z" level=info msg="Processing change to Function default/crtest" pkg=function-controller
time="2018-09-26T17:58:11Z" level=error msg="Function can not be created/updated: Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:11Z" level=error msg="Error processing default/crtest (will retry): Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:11Z" level=info msg="Processing change to Function default/crtest" pkg=function-controller
time="2018-09-26T17:58:12Z" level=error msg="Function can not be created/updated: Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:12Z" level=error msg="Error processing default/crtest (will retry): Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:12Z" level=info msg="Processing change to Function default/crtest" pkg=function-controller
time="2018-09-26T17:58:13Z" level=error msg="Function can not be created/updated: Not found a valid compilation step for c1.0" pkg=function-controller
time="2018-09-26T17:58:13Z" level=error msg="Error processing default/crtest (giving up): Not found a valid compilation step for c1.0" pkg=function-controller
ERROR: logging before flag.Parse: E0926 17:58:13.769261       1 function_controller.go:185] Not found a valid compilation step for c1.0

I have defined initcontainer images

I want to know how to test the function-controller code use gdb?

@laurencechan I have never debugged the project with GDB so I am not going to be able to help you with that. What I can tell you is that you will need to add a case in the GetCompilationContainer for the new runtime. Something like:
        case strings.Contains(runtime, "ballerina"):
                command = fmt.Sprintf("/compile-function.sh %s", funcName)
+       case strings.Contains(runtime, "c"):
+               command = fmt.Sprintf("/compile-function.sh %s", funcName)

        default:
                return v1.Container{}, fmt.Errorf("Not found a valid compilation step for %s", runtime)

Note that you need to rebuild the kubeless-controller image and use that one. You can do so executing:
make function-controller CONTROLLER_IMAGE=laurencechan/kubeless-function-controller:1

When the image is built you can edit your current deployment and change the image to see the changes.

Hi~ @andresmgot  I definitely add this

But I don't understand the make command, in which dir should i execute this command? and the kubeless-controller image, you mean  the function-controller image？

hi~@andresmgot  I have set proxy, It still stucked while RUN install_package ca-certificates

what should i do ?

It seems that the debian repositories are not reachable from within your containers. Could that be the reason?

@andresmgot  I add proxy in install_packages through dockerfile, it still didn't work

Unfortunately that step is necessary. To double check that it's a connectivity issue you can run:
docker run -it bitnami/minideb
apt-get update
apt-get install ca-certificates

That's what the install_packages wrapper does.

@andresmgot  I just did the same as you think, and commit a new image to replace the bitnami/minideb:jessie so i can delete the "RUN apt-get install ca-certificates" line in the dockerfile. as i finally got the new function controller image, how should i use it to start a new function-controller pod?

Hi~@andresmgot I already did almost the same as you said and successfully restart my new function controller, but as always, i encountered the new fail... I actually use the same image for compiling and running the webserver, but it pulled the kubeless/ccompile:1.0 success the first time and failed the second time...

If you have the image cached locally you need to set the parameter imagePullPolicy to IfNotPresent to avoid trying to pull it from dockerhub. You can use the flag --image-pull-policy=IfNotPresent or edit the running deployment kubectl edit deployment <func_name> to change it.

Hi ~@andresmgot  It woks with this flag.Thanks a lot!
But why did only the second one pull failed?both of them were presented locally。

Hi ~@andresmgot, I finally run my c function in my c runtime,but the liveness probe always failed

i defined the heahltz function like this

Hi, I am happy that the runtime is taking shape :)
I am a bit rusty about C but it seems that the connection is not being closed? To ensure that your health check is working you can open a proxy with kubectl proxy and do a request with curl to your function. Something like: curl -v http://localhost:8001/api/v1/namespaces/<func_namespace>/services/<func>:8080/proxy/healthz.
That should return a success with a header HTTP/1.1 200 OK.

Hi~ @andresmgot ,Thanks for your help, It runs well ~~

are you sure that your function is returning a response? I see in the logs you sent that the connection is closed with EOF. That usually means that the connection is closed unexpectedly. Probably your function is returning the body OK but it's not properly closing the connection.

Hi~ @andresmgot,This problem does exsits!! The record showes below

It handle the healthz and 404 well, But as for my so, It turns out to be an EOF error, But if I run the server on the lo IP and it would handle the / request well.
I load the so like this

and the error occurs on the line in the red box，If I comment this line, it works well. Then, what's wrong with it and what should I do ?

Sorry, that seems to be a runtime error of C. I am not very experienced with that language so I am not going to be able to help you with that. Maybe the logs of the pod show more information?

Cool, I am glad you figured it out

Hi~ @andresmgot

that guide is now outdated, I am afraid we should unpublished it.

that guide is now outdated, I am afraid we should unpublished it.

sorry, that guide is no longer valid, I'll remove it.

you can still autoscale functions but you cannot do it with custom metrics (you can still use memory consumption).

@andresmgot So,do you have a plan to develop the autoscale with custom metrics?

@andresmgot By the way, is the autoscale with cpu usage still available?

You can still use them, those are Kubernetes concepts that you can apply to any deployment depending on the settings of the cluster. We just need some help documenting them :)

Hi~ @andresmgotThanks for your reply~  I've tried the k8s HPA，But stucked here.

Hi~ @andresmgot  For the Cruntime  I changed and rebuild the kubeless function controller image, and the function controller keeps crashloopbackoff

Where the problem could be?

It may be related to your http_proxy configuration, you may need to whitelist the K8s API IP to be able to retrieve the kubeless configmap.

I think I had already done that.


feat: add containerd/fuse-overlayfs-snapshotter

containerd/fuse-overlayfs-snapshotter: fuse-overlayfs plugin for rootless containerd
$ aqua g -i containerd/fuse-overlayfs-snapshotter

Thanks!

@suzuki-shunsuke
I thought the "hard way" procedure would work, but I got an error with the nsenter command...
https://github.com/containerd/fuse-overlayfs-snapshotter#hard-way
$ nsenter -U --preserve-credentials -m -n -t $(cat $XDG_RUNTIME_DIR/rootlesskit-containerd/child_pid) containerd-fuse-overlayfs-grpc $XDG_RUNTIME_DIR/containerd/fuse-overlayfs.sock $HOME/.local/share/containerd-fuse-overlayfs

FATA[0000] aqua failed                                   aqua_version=1.20.2 env=linux/amd64 error="command is not found" exe_name=containerd-fuse-overlayfs-grpc program=aqua

@suzuki-shunsuke
I've been working on it for a bit, but I get an error with the nerdctl command...
https://github.com/containerd/fuse-overlayfs-snapshotter#hard-way
step1: aqua install
$ cat << '__EOF__' > aqua-local.yaml
---
# aqua - Declarative CLI Version Manager
# https://aquaproj.github.io/
packages:
- name<EMAIL_ADDRESS>- name<EMAIL_ADDRESS>- name<EMAIL_ADDRESS>- name<EMAIL_ADDRESS>- name<EMAIL_ADDRESS>- name<EMAIL_ADDRESS>__EOF__
$ aqua install --all

step2: containerd config
$ mkdir -p ~/.config/containerd
$ cat << __EOF__ > ~/.config/containerd/config.toml
version = 2
root = "$(echo $HOME)/.local/share/containerd"
state = "$(echo $XDG_RUNTIME_DIR)/containerd-rootless"

[grpc]
  address = "$(echo $XDG_RUNTIME_DIR)/containerd-rootless/containerd.sock"

[proxy_plugins]
  [proxy_plugins."fuse-overlayfs"]
    type = "snapshot"
    address = "$(echo $XDG_RUNTIME_DIR)/containerd-rootless/fuse-overlayfs.sock"
__EOF__

step3: running rootlesskit
$ rootlesskit \
  --net=slirp4netns --disable-host-loopback \
  --copy-up=/etc --copy-up=/run \
  --state-dir=$XDG_RUNTIME_DIR/containerd-rootless \
  sh -c "rm -rf /run/containerd ; sleep infinity"

step4: running containerd-fuse-overlayfs-grpc
$ nsenter -U --preserve-credentials -m -n -t $(cat $XDG_RUNTIME_DIR/containerd-rootless/child_pid) \
  sh -c "cd $(pwd) ; containerd-fuse-overlayfs-grpc $XDG_RUNTIME_DIR/containerd-rootless/fuse-overlayfs.sock $HOME/.local/share/containerd"

step5: running containerd
$ nsenter -U --preserve-credentials -m -n -t $(cat $XDG_RUNTIME_DIR/containerd-rootless/child_pid) \
  sh -c "cd $(pwd) ; containerd -c $HOME/.config/containerd/config.toml"

step6: running nerdctl
$ export CONTAINERD_SNAPSHOTTER=fuse-overlayfs
$ nerdctl -H /run/user/1000/containerd-rootless/containerd.sock run alpine

docker.io/library/alpine:latest:                                                  resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:1304f174557314a7ed9eddb4eab12fed12cb0cd9809e4c28f29af86979a3c870: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:9c6f0724472873bb50a2ae67a9e7adcb57673a183cea8b06eb778dca859181b5:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.4 s                                                                    total:  2.7 Mi (508.1 KiB/s)
FATA[0005] failed to create shim task: failed to mount rootfs component &{fuse3.fuse-overlayfs overlay [workdir=/home/takumi/.local/share/containerd/snapshots/3/work upperdir=/home/takumi/.local/share/containerd/snapshots/3/fs lowerdir=/home/takumi/.local/share/containerd/snapshots/1/fs]}: mount helper [mount.fuse3 [overlay /run/user/1000/containerd-rootless/io.containerd.runtime.v2.task/default/12a36a3820cce6ca4bb52c892fb9d951926ad4a7ac72e69f78113bd8eeb05c02/rootfs -o workdir=/home/takumi/.local/share/containerd/snapshots/3/work -o upperdir=/home/takumi/.local/share/containerd/snapshots/3/fs -o lowerdir=/home/takumi/.local/share/containerd/snapshots/1/fs -t fuse-overlayfs]] failed: "time=\"2022-10-15T18:10:11+09:00\" level=fatal msg=\"aqua failed\" aqua_version=1.20.2 env=linux/amd64 error=\"command is not found\" exe_name=fuse-overlayfs program=aqua\n": exit status 1: unknown

@suzuki-shunsuke
The nerdctl command fails, but the containerd-fuse-overlayfs-grpc command runs without problems, so I think it is OK to merge.

Released. https://github.com/aquaproj/aqua-registry/releases/tag/v3.76.0


598 - Fix pydantic validation error

Pydantic validation error happens when using IterativeGraph module.
Problem arises when the IterativeGraph module tries to use the same instantiation of a class for multiple execution, causing the class to not pass correctly llm into his functions.

Modified Search graphs to move instantiation inside IterativeGraph
IterativeGraph now creates multiple instances of the class we want to iterate over. This can be made better.

Openai models, 4o and 4o-mini, do not throw errors but also do not respect the pydantic models:
Models:
class MatchedSchool(BaseModel):
    name: str = Field(description="The name of the input candidate medical school.")
    alternate_names: List[str] = Field(description="A list of alternate names referencing this school. Could be abbreviations, or fully spelled out names, as well as names of individual departments responsible for the Medical Curriculum within the school.")
    is_med_school: Literal["true", "false"] = Field(description="Whether or not the input school is actually a medical school or has a medical program.")
    city: Optional[str] = Field(description="The city where the matched medical school campus/program facility is located, if available.")
    state: Optional[str] = Field(description="The state (or if international, the geographic/political region within the country) where the matched medical school campus/program facility is located, if available.")
    country: Optional[str] = Field(description="The country where the matched medical school campus/program facility is located, if available.")
    source: str = Field(description="Source URL where this match was found.")

class Matches(BaseModel):
    matches: List[MatchedSchool] = Field(description="A list of matched medical schools.")

Output:
{
 "name": "Centro Universit\u00e1rio Franciscano (UNIFRA)",
 "alternate_names": [
   "UNIFRA",
   "Centro Universit\u00e1rio Franciscano"
 ],
 "is_med_school": "true",
 "city": "Santa Maria",
 "state": "Rio Grande do Sul",
 "country": "Brazil",
 "sources": [
   "https://caper.ca/sites/default/files/pdf/CAPER_MedicalSchools_September_2022.xlsx",
   "https://www.facebook.com/engmatUNIFRA/",
   "https://unifra.academia.edu/ClariceMachado"
 ]
}

Gemini 1.5 flash works well with proper formatting, but gemini 1.5 pro still has the formatting issue:
File "/opt/anaconda3/envs/med_device/lib/python3.12/site-packages/langchain_core/output_parsers/json.py", line 87, in parse_result
    raise OutputParserException(msg, llm_output=text) from e
langchain_core.exceptions.OutputParserException: Invalid json output: ```json
{'matches': []}
\```

OpenAi
OpenAi behavior is expected.
FinalMode is MergeNode, its scope is to return all the collected answers plus the sources from where the answer were taken.
answer = merge_chain.invoke({"user_prompt": user_prompt})
answer["sources"] = state.get("urls")

It happened that the llm didn't returned anything under source and MergeNode added sources
You reported the same behavior in #598. Also there the "wrong" returned key was sources:

{
  "is_medical_school": true,
  "location": {
    "city": "New York",
    "state": "New York",
    "country": "United States"
  },
  "source": "https://en.wikipedia.org/w/index.php?title=New_York_College_of_Podiatric_Medicine&oldid=1219437275",
  "sources": [
    "https://en.wikipedia.org/wiki/New_York_College_of_Podiatric_Medicine"
  ]
}

Gemini
The error is not for a wrong parsing mechanism. As can be observed by your log, the llm returned an empty matches list `"matches":[]'.
This cannot be parsed with the provided schema since the schema do not allow for empty lists.
See these:
https://stackoverflow.com/questions/61468548/check-if-list-is-not-empty-with-pydantic-in-an-elegant-way
https://github.com/samuelcolvin/pydantic/issues/367

thank you


Add Request Stamping Functionality to Server and Browser SDK

Summary & Motivation
This PR adds functionality to sdk-browser and sdk-server to generate and return stamped requests of each call.
It does this by updating the codegen.js files in each package to generate stamping versions of each basic method that is effectively a call to the Turnkey API, returning the stamped request WITHOUT actually sending the request.
How I Tested These Changes
Did you add a changeset?
If updating one of our packages, you'll likely need to add a changeset to your PR. To do so, run pnpm changeset. pnpm changeset will generate a file where you should write a human friendly message about the changes. Note how this (example) includes the package name (should be auto added by the command) along with the type of semver change (major.minor.patch) (which you should set).
These changes will be used at release time to determine what packages to publish and how to bump their version. For more context see this comment.

the only ask here is that you add a corresponding changeset