[suggestion] Check face preprocessing step for all the keras face embedding models

In this repo, the face patch preprocessing step is same for all the models.
img = cv2.resize(img, target_size)
img_pixels = image.img_to_array(img)
img_pixels = np.expand_dims(img_pixels, axis = 0)
img_pixels /= 255 #normalize input in [0, 1]

Please look into it if this is valid for all, as the inference preprocessing step must match the model's training pipeline for accurate result. For example, the original Facenet repo has done it differently.

I do not understand what you mean.

I do not understand what you mean.

I have updated the questions. Kindly let me know if it requires any further clarifications.

How FaceNet does it? I could not find in that bulk file.

How FaceNet does it? I could not find in that bulk file.

I had referred to it in my question as well. Please find it here.

Here, author of the facenet declared that prewhiten makes training easier but we are not interested in training in deepface. I do not believe adding this will increase the model accuracy.

@serengil Thank you for your interest in researching it.
I was trying to point out that if we do not follow the same preprocessing step used in training the model, we will not get the expected result while inference. Suppose we train the model with just dividing the pixel values by 255 but while testing (or inference) we are not performing that operation, we cannot we expect similar results as per our validation set.
The pre and post-processing steps during model inference must be the same.
Still open to anyone's feedback on this from the community. Thank you.

@iamrishab @serengil
I am very interested in this topic and would like to discuss more about refining the pre-processing steps as well as the threshold for decisions between True and False verification.
Specifically, I've been working on creating face detection/recognition pipeline using MTCNN for detection and both VGG-Face and Facenet for calculating embeddings.
@serengil, please correct me if I'm wrong, but I believe that the True/False thresholds you have hard-coded in distance.py (and discussed at length on this post: https://sefiks.com/2020/05/22/fine-tuning-the-threshold-in-face-recognition/ ) were all decided upon from normalizing the RGB pixel values dividing by 255 (as @iamrishab mentions).
I ran some tests against 20 different people (all with between 6-30 images per person) using the default "/255" method as RGB input. I ran with both Euclidean Distance and Cosine options.
I found there was a lot of error both false positives and false negatives.
I tried to refine and found that Facenet model was originally trained on dataset where all images were normalized by calculating img RGB mean and img RGB std and then img =  (img-mean)/std
When performing this preprocess calculation (rather than simply dividing by 255 before calculating embeddings), my TRUE/FALSE verification accuracy got much better, using euclidean distance and given threshold of 10) - awesome!
Then onto VGG-Face, I found that simply dividing by 255 and using given thresholds for cosine/euclidean distance (.4 (cosine) / .55 (euclidean))  were giving me tons of false positives, and all true positives seemed to be way below the given thresholds - typically scoring at .2 or lower.
Again, I researched VGG-Face as much as I could to try and figure out how the model was trained and found some posts where it seemed that they subtracted individual R, G, and B means. (R - 131.0912, G - 103.8827 B - 91.4953) with no division, and fed those values into model: (https://github.com/ox-vgg/vgg_face2/issues/17 )  I also saw ambiguous information that the VGG-FACE model might have taken images in as BGR rather than RGB: (https://github.com/rcmalli/keras-vggface/issues/62)
So I attempted 6 different permutations of preprocessing the RGB values before sending them to embeddings, to see if there was a clearer delineation for threshold calculations.  Sadly, whether I used cosine or euclidean comparisons, I couldn't dial in the VGG-Face threshold like I seemed to accomplish with Facenet.
@serengil , I do believe deepface shines brightest if I stick with the "/255" (or any other normalization technique for that matter) and create embeddings for hundreds of thousands of people, eventually filling up a database with millions of embeddings, and then do a face recognition search of an unknown face against those millions of embeddings, that the lowest cosine variation or lowest euclidean distance will indeed have a high likelihood of matching my unknown image.
However I'd like to explore the True/False ("head to head") verification some more to try and dial in the "best embeddings" for each model.
I also noticed in your earlier blogs, you were using a keras preprocess_input function to normalize VGG faces before passing them in:  https://sefiks.com/2018/08/06/deep-face-recognition-with-keras/ - referencing the preprocess_input function in this repo file: https://github.com/keras-team/keras-applications/blob/master/keras_applications/imagenet_utils.py But currently in deepface repo it doesn't seem we are still using that tool?  In functions.py we are including "from tensorflow.keras.applications.imagenet_utils import preprocess_input" but doesn't appear we are calling it anywhere?
I'm sure there are still complex elements that I'm failing to fully understand (Was this VGGFace model trained on RESNET50 or "vgg16" or "senet50" or something else?  Where did the vgg_face_weights.h5 file come from?)  but yes let's open conversation so we can fully understand it because it seems there is confusion in many threads around the internet.
Thanks much, sorry for the long post.

@trevorgribble thank you for this informative message. I reopened this issue.
It seems that especially in facenet preprocessing increases the model accuracy dramatically based on your experiments. Right?
Do you have the code snippet for preprocessing?

For reference, pls follow this link. Ref
The following is valid for model trained on VGGFace2 dataset.
The below preprocessing script is used during facenet 2017 model training which gived 128d embeddings. The model is trained with criteria of minimizing euclidian distance between two vectors.
def prewhiten(x:np.ndarray):
      """Normalzing the face patch"""
      mean = np.mean(x)
      std = np.std(x)
      std_adj = np.maximum(std, 1.0 / np.sqrt(x.size))
      y = np.multiply(np.subtract(x, mean), 1 / std_adj)
      return y

For facenet 2018 model which gives 512d embeddings. Criteria is minimizing cosine distance between two vectors.
def standardization(x:np.ndarray):
    """Standard the face patch normalization"""
    return (np.float32(x) - 127.5) / 128.0

@serengil
I made a handful of code changes to suit my personal use cases, and basically removed the code of chunk:
img_pixels = image.img_to_array(img)
img_pixels = np.expand_dims(img_pixels, axis = 0)
img_pixels /= 255 #normalize input in [0, 1]

out of preprocess_face and moved it into DeepFace.py represent() function after the call to preprocess_face.
I added a new variable called "normalization" to represent() to try different techniques for VGG-Face and look at output variations in cosine/euclidean to see if there's a difference. (I haven't taken this step as far as you did in the "refining the threshold" post, mapping all the graphs, but I would imagine this would be the next step.)
Note I also removed the line "img_pixels = image.img_to_array(img)" altogether throughout the code base because I couldn't figure out why we needed to use that extra dimension of array.
def represent(img_path, model_name = 'VGG-Face', model = None, normalization="v1", enforce_detection = False, detector_backend = 'mtcnn', align = True):
...
...
...

img = img.astype('float64')

#Facenet case works well (as @iamrishab points out this aligns with the prewhiten function for the 128 embeddings)
if(model_name=='Facenet'):
                img = np.expand_dims(img, axis = 0)
                mean, std = img.mean(), img.std()
                img = (img-mean)/std
#Still trying to figure out optimal VGG-Face preprocess steps (I added a normalization variable that I passed in to try and compare different styles of preprocessing based on my above post, but still haven't come across optimal case. Perhaps we can collaborate in detail)
elif (model_name=='VGG-Face'):
		if (normalization == "v1"):
			#BGR mean subtraction / 255 normalization
			img[..., 0]-= 131.0912
			img[..., 1] -= 103.8827
			img[..., 2] -= 91.4953
			img = img[..., ::-1]
			img /=255
		elif(normalization =="v2"):
			#RGB mean subtraction / 255 normalization
			img[..., 0]-= 131.0912
			img[..., 1] -= 103.8827
			img[..., 2] -= 91.4953
			img /=255
		elif(normalization =="v3"):
			#BGR mean subtraction normalization
			img[..., 0]-= 131.0912
			img[..., 1] -= 103.8827
			img[..., 2] -= 91.4953
			img = img[..., ::-1]
		elif(normalization =="v4"):
			#RGB mean subtraction normalization
			img[..., 0]-= 131.0912
			img[..., 1] -= 103.8827
			img[..., 2] -= 91.4953
		elif(normalization=="v5"):
			# simply / 255 (as was case in original @serengil code)
			img /=255
		elif(normalization=="v6"):
			# simply / 127.5 - 1 (similar to facenet 2018 model preprocessing step as @iamrishab posted)
			img /= 127.5
			img -= 1
else:  (for all other models besides Facenet and VGG-Face, I'm keeping original code as I haven't gotten into it yet)
		img = np.expand_dims(img, axis = 0)
		img /= 255 #normalize input in [0, 1]

Thank you a lot. I will move your logic in the next release.

@iamrishab between my experiements and confirming with your link above, it appears we have a very good lead on the preprocess/normalization steps before passing faces into the Facenet 2017 model that is included in deepface (with 128d embeddings)  (which as you linked above was trained on the VGG2Face Dataset).

@serengil You're very welcome. Thanks for all your work on this repo and all associated blog posts and videos.  Image preprocessing consistency across both model training and model inference has always been the most confusing and challenging stage of all deep learning tools I have investigated, and I've seen accuracy improve tremendously when it all lines up....thus my dedication to making sure we truly get this as accurate as possible and eliminate any ambiguities if possible.
As I dive more and more into researching how VGGFace was preprocessed before training, I note there are 2 different models: VGGFace and VGGFace2.
To be sure, is your VGGFace.py file creating the VGGFace or the more recent VGGFace2 model in keras?
https://github.com/serengil/deepface/blob/024fe6843fd12eea63f06ac315ac9c4cd98c0094/deepface/basemodels/VGGFace.py
Info on both models available here: https://machinelearningmastery.com/how-to-perform-face-recognition-with-vggface2-convolutional-neural-network-in-keras/
My thinking is original VGGFace was trained on 2622 people and VGGFace2 was trained on 9131 people, so the model we are using here aligns with VGGFace (original).

the output layer of the vgg-face model has 2622 nodes, so, it was trained on vggface data set instead of vggface2

the output layer of the vgg-face model has 2622 nodes, so, it was trained on vggface data set instead of vggface2

This is something I was also curious about: would it be possible to implement a VGGFace2 model as well? In addition, it might be helpful in the documentation to clarify and/or cite papers for each implementation to better understand exactly what deepface is providing.
Independent of this, thank you @serengil for providing this extremely convenient interface to these models!

the output layer of the vgg-face model has 2622 nodes, so, it was trained on vggface data set instead of vggface2

This is something I was also curious about: would it be possible to implement a VGGFace2 model as well? In addition, it might be helpful in the documentation to clarify and/or cite papers for each implementation to better understand exactly what deepface is providing.
Independent of this, thank you @serengil for providing this extremely convenient interface to these models!

It would be possible, by retraining the same model with the VGGFace2 dataset. But right now the original dataset has been taken down by the author.
Luckily, a pretrained Keras model trained on the dataset was made available by the author here : https://drive.google.com/file/d/1AHVpuB24lKAqNyRRjhX7ABlEor6ByZlS/view

Does facenet not require RGB?
You convert image BGR2RGB before pass to MTCNN and RatinaFace but why not Facenet?
In my case I observe facenet gives better results when I pass RGB image instead of BGR.

Does facenet not require RGB?
You convert image BGR2RGB before pass to MTCNN and RatinaFace but why not for Facenet?
In my case I observe facenet gives better results when I pass RGB image instead of BGR.

#Still trying to figure out optimal VGG-Face preprocess steps (I added a normalization variable that I passed in to try and compare different styles of preprocessing based on my above post, but still haven't come across optimal case. Perhaps we can collaborate in detail)
elif (model_name=='VGG-Face'):
if (normalization == "v1"):
#BGR mean subtraction / 255 normalization
img[..., 0]-= 131.0912
img[..., 1] -= 103.8827
img[..., 2] -= 91.4953
img = img[..., ::-1]
img /=255
elif(normalization =="v2"):
#RGB mean subtraction / 255 normalization
img[..., 0]-= 131.0912
img[..., 1] -= 103.8827
img[..., 2] -= 91.4953
img /=255
elif(normalization =="v3"):
#BGR mean subtraction normalization
img[..., 0]-= 131.0912
img[..., 1] -= 103.8827
img[..., 2] -= 91.4953
img = img[..., ::-1]
elif(normalization =="v4"):
#RGB mean subtraction normalization
img[..., 0]-= 131.0912
img[..., 1] -= 103.8827
img[..., 2] -= 91.4953
elif(normalization=="v5"):
# simply / 255 (as was case in original @serengil code)
img /=255
elif(normalization=="v6"):
# simply / 127.5 - 1 (similar to facenet 2018 model preprocessing step as @iamrishab posted)
img /= 127.5
img -= 1
else:  (for all other models besides Facenet and VGG-Face, I'm keeping original code as I haven't gotten into it yet)
img = np.expand_dims(img, axis = 0)
img /= 255 #normalize input in [0, 1]

@trevorgribble I actually have a question on the normalization part for VGGface. I know that this is what was recommended by the original authors, but I don't see the logic behind subtracting the dataset RGB mean from the data in the inference stage. I might be missing something, but the normalization step seems to suggest the assumption that the dataset where the model was trained is representative of the population distribution. Thus, for inferences of unseen images, we need to normalize them by the original dataset mean.
My thoughts on this would be to remove the normalization part where rgb means are subtracted when re-training the VGGface model and just stick to the normalizing to 0-1 range. Another way would be to rely on the default preprocessing step by Keras: https://www.tensorflow.org/api_docs/python/tf/keras/applications/vgg16/preprocess_input, in this case, they simply divided the image pixels by 127.5 and then normalized to -1 to 1 range:
if mode == 'tf':
    x /= 127.5
    x -= 1.
    return x

I'm not an expert but I'm very interested in people's thoughts on the RGB normalization for vggface. Thanks!

@serengil would it be possible to use the VGGface 2 model with the 512 dimensions? the Pre- trained model is already available here https://github.com/davidsandberg/facenet
i tried to replace your facenet_weights.h5 file with the new pretrained model but i ran into several errors (i changed file names in facenet.py) but seems like i am not able to read the file.
any idea how to load this pre-trained model with 512 dimensions ?

@sadimoodi sure but could you open a custom issue about this. We can follow it there.

@serengil  was preprocessing improved according to this post in the new release?

@sadimoodi no, I'm going to inform here and close this post when it is fixed.

Just out of curiosity, would it take a long time for the release? I badly need to use this in my research.

@trevorgribble thank you for this informative message. I reopened this issue.
It seems that especially in facenet preprocessing increases the model accuracy dramatically based on your experiments. Right?
Do you have the code snippet for preprocessing?

Is it included in the current release?

not yet @Akila-Ayanthi

Just out of curiosity, will it take a long time? I want to use this for my research.

I focused on some other issues and that's why this issue is still alive. I plan to focus on this in the next couple of days.

I focused on some other issues and that's why this issue is still alive. I plan to focus on this in the next couple of days.

Thanks @serengil for all your amazing work!

Okay, looking forward to these changes. Thank you @serengil.

Thank you a lot. I will move your logic in the next release.

Is this done?

@nbhupendra I'm going to close this issue when it is done.

deepface 0.0.66 is on live!
I added a normalization argument in verify, find and represent function. Its default value is base and it will work as is in this case.
If you set Facenet, v1, v2, ... to normalization, it will apply the logic @trevorgribble shared.
@trevorgribble buddy, I mentioned you in the source code as well. Thank you!

the output layer of the vgg-face model has 2622 nodes, so, it was trained on vggface data set instead of vggface2

@serengil and @trevorgribble: If the deepface VGG-Face model is version 1, then the values being subtracted for the various normalization options (i.e., v1, v2, v3, and v4) added to version 0.0.66 are incorrect since those are from VGGFace2. The correct values for both versions can be found here in the keras-vggface repository:
https://github.com/rcmalli/keras-vggface/blob/master/keras_vggface/utils.py#L31
Note that the logic itself is slightly different than what @trevorgribble proposed. I'm currently working on testing how using the VGGFace1 values may improve the VGG-Face model results. It's worth noting that the "Facenet" normalization approach did improve the results in my testing using the lfw pairs pulled from sklearn (i.e., the "test" subset).

I have been able to replicate the keras_vggface preprocessing using a slightly modified version of deepface. The revised version is here, and the only change is to the new normalize_input function from 0.0.66:
def normalize_input(img, normalization = 'base'):

	#issue 131 declares that some normalization techniques improves the accuracy

	if normalization == 'base':
		return img
	else: #@trevorgribble recommend the following idea (further edits by @davedgd)

		img *= 255 #restore input in scale of [0, 255] because it was normalized in scale of  [0, 1] in preprocess_face

		if normalization == 'Facenet':
			mean, std = img.mean(), img.std()
			img = (img - mean) / std

		elif normalization == 'VGGFace-v1':
			# mean subtraction based on VGGFace1 training data
			img[..., 0] -= 93.5940
			img[..., 1] -= 104.7624
			img[..., 2] -= 129.1863

		elif(normalization == 'VGGFace-v2'):
			# mean subtraction based on VGGFace2 training data
			img[..., 0] -= 91.4953
			img[..., 1] -= 103.8827
			img[..., 2] -= 131.0912

		elif(normalization== 'tf'): # normalize between -1 and 1
			# simply / 127.5 - 1 (similar to facenet 2018 model preprocessing step as @iamrishab posted)
			img /= 127.5
			img -= 1

	#-----------------------------

	return img

Additionally, I wrote some code to check this against keras_vggface. You can run this on colab (just use one of the images from deepface to test):
!pip install keras_vggface keras_applications https://github.com/davedgd/deepface/archive/refs/heads/master.zip --no-cache-dir --quiet

from keras_vggface import utils
from deepface.commons import functions

import numpy as np
from matplotlib import pylab as P

theImage = './img1.jpg'

# keras_vggface
im = functions.preprocess_face(theImage, align = False, enforce_detection = False)[0]
keras_vgg = im[..., ::-1] * 255
keras_vgg = utils.preprocess_input(keras_vgg, version=1)

# deepface (base)
im = functions.preprocess_face(theImage, align = False, enforce_detection = False)[0]
df_base = functions.normalize_input(im, normalization = 'base')

# deepface (VGGFace-v1)
im = functions.preprocess_face(theImage, align = False, enforce_detection = False)[0]
df_vgg = functions.normalize_input(im, normalization = 'VGGFace-v1')

# plots
f, ax = P.subplots(1, 4)
ax[0].imshow(functions.preprocess_face(theImage, align = False, enforce_detection = False)[0])
ax[1].imshow(keras_vgg)
ax[2].imshow(df_base)
ax[3].imshow(df_vgg)

print('keras_vggface and deepface with VGGFace-v1 preprocessing identical:', np.array_equal(keras_vgg, df_vgg))

If you run the code, you will see both packages return the same result now (i.e., identical matrices and images post-preprocessing).
In terms of performance metrics, it's a bit difficult to comment on it since these changes drastically change some of the distance_metric results (although this is primarily true for euclidean but not cosine and euclidean_l2). It will be necessary to recalculate the thresholds to see if this approach improves the result overall, but in my initial testing -- at least for cosine and euclidean_l2 -- it does not seen to harm it (e.g., VGGFace-v1 relative to base normalization)...

deepface 0.0.66 is on live!
I added a normalization argument in verify, find and represent function. Its default value is base and it will work as is in this case.
If you set Facenet, v1, v2, ... to normalization, it will apply the logic @trevorgribble shared.
@trevorgribble buddy, I mentioned you in the source code as well. Thank you!

@serengil for Facenet512 which normalization has to apply? v6 or someother?

@davedgd wrapped vgg-face model was not trained on vgg-face2 dataset. So, you should you v1 normalization.
@nbhupendra, I think that it should be Facenet normalization as well.

@davidwdw, I will add vgg-face-v1 and v2 normalizations tonight in the source code and pip

@davidwdw, I will add vgg-face-v1 and v2 normalizations tonight in the source code and pip

@serengil: thanks for answering my question about v1 and v2 as well. Hopefully my code/links were helpful.
I did a bit of additional research on ArcFace and found one more preprocessing possibility described in the paper under the section named "Input Setting":

Following [46, 23], we use five facial landmarks (eye centres, nose tip and mouth corners) [49] for similarity transformation to normalise the face images. The faces are cropped and resized to 112×112, and each pixel (ranged between [0,255]) in RGB images is normalised by subtracting 127.5 then divided by 128.

According to various GitHub posts however, it's unclear if it's necessary or if the raw/unstandardized values should be used instead, e.g., Question about preprocessing steps, Preprocessing the face image thumbnail, Dose it need to substract 127.5 and divde 128.
In light of that, I'd suggest two more variations to the normalization function: ArcFace and raw. I've added them below as a suggestion in case it's helpful:
def normalize_input(img, normalization = 'base'):

	#issue 131 declares that some normalization techniques improves the accuracy

	if normalization == 'base':
		return img
	else: #@trevorgribble recommend the following idea (further edits by @davedgd)

		img *= 255 #restore input in scale of [0, 255] because it was normalized in scale of  [0, 1] in preprocess_face

		if normalization == 'Facenet':
			mean, std = img.mean(), img.std()
			img = (img - mean) / std

		elif normalization == 'VGGFace-v1':
			# mean subtraction based on VGGFace1 training data
			img[..., 0] -= 93.5940
			img[..., 1] -= 104.7624
			img[..., 2] -= 129.1863

		elif(normalization == 'VGGFace-v2'):
			# mean subtraction based on VGGFace2 training data
			img[..., 0] -= 91.4953
			img[..., 1] -= 103.8827
			img[..., 2] -= 131.0912

		elif(normalization == 'ArcFace'):
			img -= 127.5
			img /= 128

		elif(normalization == 'tf'): # normalize between -1 and 1
			# simply / 127.5 - 1 (similar to facenet 2018 model preprocessing step as @iamrishab posted)
			img /= 127.5
			img -= 1

		elif(normalization == 'raw'):
			# return original pixel values multiplied by 255 but unscaled
			pass

	#-----------------------------

	return img

@trevorgribble and @ davedgd thank you again
deepface 0.0.67 is on live and now it is simplified.
def normalize_input(img, normalization = 'base'):

	#issue 131 declares that some normalization techniques improves the accuracy

	if normalization == 'base':
		return img
	else:
		#@trevorgribble and @davedgd contributed this feature

		img *= 255 #restore input in scale of [0, 255] because it was normalized in scale of  [0, 1] in preprocess_face

		if normalization == 'raw':
			pass #return just restored pixels

		elif normalization == 'Facenet':
			mean, std = img.mean(), img.std()
			img = (img - mean) / std

		elif(normalization=="Facenet2018"):
			# simply / 127.5 - 1 (similar to facenet 2018 model preprocessing step as @iamrishab posted)
			img /= 127.5
			img -= 1

		elif normalization == 'VGGFace':
			# mean subtraction based on VGGFace1 training data
			img[..., 0] -= 93.5940
			img[..., 1] -= 104.7624
			img[..., 2] -= 129.1863

		elif(normalization == 'VGGFace2'):
			# mean subtraction based on VGGFace2 training data
			img[..., 0] -= 91.4953
			img[..., 1] -= 103.8827
			img[..., 2] -= 131.0912

		elif(normalization == 'ArcFace'):
			#Reference study: The faces are cropped and resized to 112×112,
			#and each pixel (ranged between [0, 255]) in RGB images is normalised
			#by subtracting 127.5 then divided by 128.
			img -= 127.5
			img /= 128

	#-----------------------------

	return img

As mentioned, you should look at distance instead of verified key because thresholds should be tuned for each normalization form. On the other hand, I already tuned them for base normalization. So, you can trust verified key in the response if you are using base normalization.

Thank you.

@trevorgribble and @ davedgd thank you again
deepface 0.0.67 is on live and now it is simplified.
def normalize_input(img, normalization = 'base'):

	#issue 131 declares that some normalization techniques improves the accuracy

	if normalization == 'base':
		return img
	else:
		#@trevorgribble and @davedgd contributed this feature

		img *= 255 #restore input in scale of [0, 255] because it was normalized in scale of  [0, 1] in preprocess_face

		if normalization == 'raw':
			pass #return just restored pixels

		elif normalization == 'Facenet':
			mean, std = img.mean(), img.std()
			img = (img - mean) / std

		elif(normalization=="Facenet2018"):
			# simply / 127.5 - 1 (similar to facenet 2018 model preprocessing step as @iamrishab posted)
			img /= 127.5
			img -= 1

		elif normalization == 'VGGFace':
			# mean subtraction based on VGGFace1 training data
			img[..., 0] -= 93.5940
			img[..., 1] -= 104.7624
			img[..., 2] -= 129.1863

		elif(normalization == 'VGGFace2'):
			# mean subtraction based on VGGFace2 training data
			img[..., 0] -= 91.4953
			img[..., 1] -= 103.8827
			img[..., 2] -= 131.0912

		elif(normalization == 'ArcFace'):
			#Reference study: The faces are cropped and resized to 112×112,
			#and each pixel (ranged between [0, 255]) in RGB images is normalised
			#by subtracting 127.5 then divided by 128.
			img -= 127.5
			img /= 128

	#-----------------------------

	return img

As mentioned, you should look at distance instead of verified key because thresholds should be tuned for each normalization form. On the other hand, I already tuned them for base normalization. So, you can trust verified key in the response if you are using base normalization.

Hi @sadimoodi Thanks for quick update. Can you please let me know when to choose Facenet2018.  in case of Facenet and facenet512?

Hi @serengil Thanks for quick update. Can you please let me know when to choose Facenet2018. in case of Facenet and facenet512?

@serengil any suggestion on this?

@nbhupendra the both works. but facenet overperforms than facenet2018.

I have found that enabling “Facenet” normalization, when using the “Facenet512” model and “euclidean_l2” distance metric, actually decreased the accuracy. I.e the resultant Euclidean distance of the two embeddings of Angelina Jolie was larger. Is this expected?


pymapdl run (vm2) creates emtpy vrt result file

I am trying to run a VM to test the pymapdl converter: pymapdl.convert_script
If I run these VM files, they write out results to  a .vrt file.
i.e.: mapdl.run("/OUT,vm2,vrt")
The script is based on vm2: https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v212/en/ans_vm/Hlp_V_VM2TXT.html
That file is empty in the directory where I launch the analysis.
I set the APDL run to that directory at the top of the script.
local = os.getcwd()
setLocal = "/CWD," + local
mapdl.run(setLocal)
Expected is the .vrt file to not be empty. I found the successful vrt in my AppData directory(?) from an earlier run.
VM2.py.txt
vm2success.vrt.txt
vm2unsuccess.vrt.txt

I'm closing this.


Dask 2024.8 started failing when indexing result of numpy.flatnonzero

Describe the issue:
scikit-image's CI started failing with Dask 2024.8 when it worked fine before with 2024.7. As described in https://github.com/scikit-image/scikit-image/issues/7491, we traced the failure
ValueError: Array chunk size or shape is unknown.

to a line that tries to index the result of a numpy.flatnonzero operation. See the MCVE below which fails for 2024.8 but works fine for 2024.7.
Minimal Complete Verifiable Example:
import numpy as np
import dask.array as da

hist = da.from_array(np.arange(256, dtype=int), chunks=(256,))
result = np.flatnonzero(hist)
result[[0, -1]]  # ValueError: Array chunk size or shape is unknown.

Anything else we need to know?:
From reading the docs that are pointed out by the error message, it seems this behavior might not be entirely unexpected and this might not actually be a bug. However, I'm unsure why it worked before with 2024.7. And since I currently don't really now how to address this on scikit-image's side without adding an extra code path to support Dask 2024.8, I'd be grateful for feedback and context.
Sorry if I missed any existing issues or docs that already describe or explain this.
Environment:

Dask version: 2024.8.0
Python version: 3.12.0
Operating System: Linux 6.10.3-arch1-1 (64-bit)
Install method (conda, pip, source): pip

Thanks for the report.
Yeah, unknown chunks are annoying. We can indeed make this work, since there is only a single chunk along the axis with unknown chunks, but we would still raise if you have more than one chunk there. The previous implementation actually let this through, but returned incorrect results, i.e.
hist = da.from_array(np.arange(256, dtype=int), chunks=(128, ))
result = np.flatnonzero(hist)
result[[0, -1]].compute()

this returned
 array([  1, 127])

which is obviously incorrect. So your case will work again, but as soon as you have more than one chunk, it will continue to raise. computing the chunk sizes is the solution for this case

We've decided to remove the unofficial dask support & test's for now from scikit-image. Thanks for the feedback @phofl!

Just to put a finer point on it: we removed the failing test, we did not change any of the code paths that previous allowed dask arrays through (they still do).

We are moving away from implicit computes in Dask, this can have heavy performance penalties, so it's unlikely that we will add that back in again. This should be an explicit call by users (scikit-image in this case), but I can understand if you don't necessarily want to do this.

Perhaps this is the more relevant question for us: does dask (aim to) support the Array API? If so, that is a path that skimage can follow, but it still would not involve explicit dask-specific code such as compute_chunk_sizes.
If the decision here implies Array API incompatibility, then that should raise some warning flags.

Dask is explicitly mentioned as a target at https://data-apis.org/array-api/latest/
However, the following code no longer works with dask:
import numpy as np
import dask.array as da
import array_api_compat

d = da.from_array(np.arange(256), chunks=(12,))
nx = nx = array_api_compat.get_namespace(d)
nx.flatnonzero(d)[[0, -1]].compute()

Dask was also one of the examples used in Array Programming with NumPy.
I am not yet that familiar with the Array API, so I will chat to @lucascolley here at EuroSciPy to find out whether my understanding is correct, and perhaps @asmeurer to help me understand whether https://github.com/data-apis/array-api-compat does, as the README seems to indicate, fully support Dask.

This looks unrelated to array-api-compat to me - the following also fails:
import dask.array as da
import numpy as np
d = da.from_array(np.arange(256), chunks=(12,))
da.flatnonzero(d)[[0, -1]]

Note that flatnonzero is not in the standard, so this seems to be just a Dask regression (perhaps deliberate?).

Should nx then have a flatnonzero member?

I think you are looking for this part of the array-api-compat docs:

Avoid Restricting Behavior that is Outside the Scope of the Standard. All array libraries have functions and behaviors that are outside of the scope of what is specified by the standard. These behaviors should be left intact whenever possible, unless the standard explicitly disallows something. This means

All namespaces are extended with wrapper functions. You may notice the extensive use of import * in various files in array_api_compat. While this would normally be questionable, this is the one actual legitimate use-case for import *, to re-export names from an external namespace.
All wrapper functions pass **kwargs through to the wrapped function.
Input types not supported by the standard should work if they work in the underlying wrapped function (for instance, Python scalars or np.ndarray subclasses).

By keeping underlying behaviors intact, it is easier for libraries to swap out NumPy or other array libraries for array-api-compat, and it is easier for libraries to write array library-specific code paths.

This would make sense to me for a library that explicitly supports the Array API, but I don't think dask does (there's no __array_namespace__), and therefore I suppose it becomes array-api-compats responsibility to filter out functions that don't adhere to the specification, or won't work when called from nx.*.

First of all, please note that the above example only worked in the past because there was only one chunk. As soon as you move to a multi chunk array, you saw the same exception on older dask versions, i.e. this never actually worked. Real dask arrays always have more than one chunk. I strongly recommend to use multi chunked arrays (two are typically sufficient) if you write any dask tests.

Perhaps this is the more relevant question for us: does dask (aim to) support the Array API?

There hasn't been an explicit decision about this. I'm not aware of any dask maintainer who is working on compatibility to Array API and this is certainly not something that will come for free. While dask is being mentioned on the Array API documentation page, I do not know which (if any) maintainer participated in the definition of that spec. To the best of my knowledge, we are currently not fully compatible and unless somebody is stepping up and is willing to invest serious time into this, I doubt it is going to happen.
So far, when we encountered requests or issues raised about Array API compatibility, this often affected cases that are notoriously hard to deal with using a distributed or lazy array.
Compatibility alone might be achievable but in our experience these implicit compute patterns are strong anti patterns [1] that are surprising people and are actively harmful in many cases. We are preferring a non surprising and easy to use API and are willing to sacrifice compatibility for this if necessary.
Note, that this kind of problem is even explicitly mentioned (to some extent) in the Array API about lazy implementations, see https://data-apis.org/array-api/latest/design_topics/lazy_eager.html
I'm also not deeply familiar with the Array API spec but as the above page already mentions, Array API is not trying to deal with the different semantics around lazy arrays and I'm not sure where this is actually breaking down.
[1] Think about an array of a couple TB in a remote storage that is being loaded entirely just to allow simple indexing as shown in this example. If we called compute_chunk_sizes ourselves, all of this would happen (and cost money) just because a user did arr[[0, 1]]

it becomes array-api-compats responsibility to filter out functions that don't adhere to the specification

flatnonzero is not in the standard, so I don't think this counts as 'not adhering to the specification'. From the spec:

A conforming implementation of the array API standard must provide and support all the functions, arguments, data types, syntax, and semantics described in this specification.
A conforming implementation of the array API standard may provide additional values, objects, properties, data types, and functions beyond those described in this specification.

Just to be clear, there have been Dask developers involved in the array API (for instance, @jakirkham). There have been discussions about adding array API support to Dask itself (most notably at https://github.com/dask/dask/pull/8750, and there's also https://github.com/dask/dask/issues/10994). For now, array-api-compat wraps libraries like Dask that are close to the array API but have various differences. Lazy arrays are definitely something that we care about in the array API and have tried to make allowances for. For instance, APIs in the standard that have value-dependent shapes are optional (like boolean indexing and unique). If you feel that there are issues with the array API that would prevent Dask from adopting it, please bring them up in the array-api repo.


Autocomplete username is broken in IE8

The autocompletion of usernames in IE8 puts the autocompleted name at the beginning of the new-message input regardless of where the @ token was.

David, can you close this issue?


[fix] mongoid cursor error when executing site copy

Coverage increased (+0.02%) to 82.802% when pulling 1328a93d84ee02cb8550cdd43b24215cffac26dc on sunny4381:mongoid-cursor-error into 16a8c4eacdc9beb2afba9b0a78eb430fc1ced21c on shirasagi:master.


Measure Spec Export Destroys Autolayout's Stacked Groups

On a lot of occasions pressing Spec export, completely resizes, moves, distorts Stacked Groups.
It seems like it's happening more if stacked layers have stacked layers inside.
It didn't happen before sketch 49 or so.

Same problem here - elements stacked and/or positioned with Auto-Layout after exporting measurements to html are reordered or in different positions. Thanks for help in advance! :)

After testing a little bit more, it seems it has to do with symbols that are contained inside Stacked groups.
If i turn off Anima plugin the export is good, so i don't know if it's the new Sketch, Anima, or Sketch-measure that causing the issue.
But you can see sketch-measure destroying your groups while it's exporting, perhaps when it ungroups everything one by one for the export.

This is a relevant page from Anima:
https://blog.animaapp.com/with-great-power-ed8bb3dc5e4d

'Prepare Export to other plugins' from Anima Toolkit seems to do the trick, and faster than manually ungrouping things yourself

It doesn't solve the issue for me though, cause it leaves Stacked groups on. And my problem lies there.

Sorry to hear you're still having trouble. For me, 'Prepare Export to other plugins' disables the dynamic stacking function while retaining the position. This being the issue, it's exported fine for me. You're using this feature on the Anima plugin correct?

I tried running Anima Toolkit's 'Prepare Export to other plugins' command multiple times, but it didn't work for me. Eventually I turned off the Anima Toolkit plugin, restarted Sketch, and exported the specs with no issues.

This is still an issue now from arguably the most popular and useful sketch plugin makes sketch measure basically unusable without the pain of disabling another plugin.... can we fix this? I would happily pay money to have this fixed <3


Update badges

pypip is down (related: https://github.com/badges/pypipins/issues/37), so I updated badges to use img.shields.ios which is online :)
Also delete downloads badge, because there info is not public (related: https://stackoverflow.com/a/38102521/559282)

Thanks!


Secrets don't work

Component
server
Describe the bug
Pipeline doesn't start with error secret "secret_name" not found or not allowed to be used.

Create pipeline:

steps:
  test:
    image: alpine
    commands:
      - printenv
    secrets:
      - test_secret

Create secret:

woodpecker=# select * from secrets;
 secret_id | secret_org_id | secret_repo_id | secret_name |      secret_value       | secret_images |   secret_events
-----------+---------------+----------------+-------------+-------------------------+---------------+-------------------
         5 |             0 |              1 | test_secret | value_test_secret_value | ["alpine"]    | ["manual","push"]
(1 row)

Run pipeline manually.
Get error:

System Info
2.0.0-rc.0, next-342b25826c, Kubernetes

Additional context
woodpecker-server-rc.log
woodpecker-server-342b25826c.log
Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

If you set an image the secret is only available to plugins, so not for this one as you set commands


fix(stagger): add leave animation stagger

add missing animation stagger for au-stagger-leave
fix stagger animation tests
https://github.com/aurelia/animator-css/pull/60#issuecomment-331381478

Thanks for the contribution. I'm not next to a PC but will take a look at it on Monday.

@z-brooks great work thanks


Wait for thrift port to open before connecting

Makes sense to me - we start the Native Server before the Thrift Server, so that could definitely race.

Merging with the "lgtm" from joel, and an offline "+1" from Hatch.

belated +1

Turns out Russ didn't give me a +1, and I just can't read offline messages properly. Alas.


Document Default/Generic Resiliency Policies

What content needs to be created or modified?
1.9 will introduce support for default/generic resiliency policies and they will need to be documented.
Describe the solution you'd like
Document the default policies, how to set them, and where they fit into the overall resiliency story.
Where should the new material be placed?
https://docs.dapr.io/operations/resiliency/policies/
The associated pull request from dapr/dapr, dapr/components-contrib, or other Dapr code repos
https://github.com/dapr/dapr/pull/4906
Additional context
There will be an additional PR for components later, but they can all be handled under this docs issue.

looks like this was resolved - there is a separate issue for adding resiliency specs to each component, so closing this.


What do the Run 1 lensed quasars look like?

Hey @jbkalmbach did you see @jchiang87 's postage stamp code in #149 ? Since you know the lens RAs and dec's, I guess you could start pulling out our targets from @TomGlanzman 's images ( #137 ) and displaying their images. Show us some lenses! :-)

I started to look at the images on the data catalog online but wasn't getting what I expected when I tried to make postage stamps. So, I looked at the centroid files and noticed that the ids for the sprinkled AGN from the instance catalogs are not in the 'SourceID' column in the centroid files. There are some 'SourceID' values that look similar but do not match the instance catalog. Does anybody have any idea why this might be?

Hmmm. This does not sound good! Are the supernovae IDs in the centroid
files?
On Tue, Mar 8, 2016 at 4:39 PM, Bryce Kalmbach<EMAIL_ADDRESS>wrote:

I started to look at the images on the data catalog online but wasn't
getting what I expected when I tried to make postage stamps. So, I looked
at the centroid files and noticed that the ids for the sprinkled AGN from
the instance catalogs are not in the 'SourceID' column in the centroid
files. There are some 'SourceID' values that look similar but do not match
the instance catalog. Does anybody have any idea why this might be?
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/155#issuecomment-194038645
.

Rahul and I took a look at this and it seems that it might be caused by a precision error when phosim converts all ids to a float. We looked at a small example phosim output and found that the id of an object was off by a small integer number of values in the centroid file from the phosim catalog.

However I was able to use @jchiang87 to get a postage stamp image of where a lens system is located. Here it is! Obviously not ideal, but definitely gives us something to talk about in the morning.

However, one comment about the postage stamps I want to make is that it should at least be centered on the bright part of the image since I gave it the catalog values for the lens galaxy, but it seems to be off. Is there a reason why the coordinates from the catalog and from the image might be slightly different?

Here's another fun one:

I SEE AGN TWICE! :-) :-) :-) See how much more fun this project is with
lensed quasars than without? I was putting a brave face on it yesterday
evening. Well done for tracking them down, Bryce.
There should be some fainter systems where the lens galaxy is bright and
visible (maximal OM10 velocity dispersion at z > 0.2) compared to the faint
AGN images (high OM10 image magnitudes)  - I'd go after some of those next.
A gallery of color images would be really helpful - the red colors of the
lenses should pop out.
On Wed, Mar 9, 2016 at 2:09 AM, Bryce Kalmbach<EMAIL_ADDRESS>wrote:

Here's another fun one:
[image: stamp_1]
https://cloud.githubusercontent.com/assets/4755090/13631903/e8dede00-e59b-11e5-9b32-a859240dcf41.png
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/155#issuecomment-194221205
.

@jbkalmbach If you are just using the WCS that comes from phosim, I don't think it's good to more than a few tens of pixels.  We didn't turn of tracking errors, so there is some wobble in the boresite.  That may be what you are seeing.  It should get much better after running through the DM pipelines.

@jbkalmbach Also worth noting that the InstanceCatalog RA and Dec are unrefracted.  PhoSim applies atmospheric refraction, which will be another shift in object positions.

Ok, as these were single epoch images I will check for consistency in the coadds. @danielsf does that get taken care of in dm processing?

@jbkalmbach @danielsf I have a feeling this issue could be closed, if the problems identified are correctly issued elsewhere and referred to here. Can you make sure of this and then shut this down please, Bryce? Thanks!

Yes, I think this can be closed. Problems from this are related to issues #175 and #168 and can be discussed there.


Get the element's position relative to the boundary when it is being dragged

At the moment, when an element is dragged the ngxDragged event emits an NgxDrag object which contains the native element and the object position in the viewport.
Is there any way to get the element's relative position to the boundary?

At the moment, when an element is dragged the ngxDragged event emits an NgxDrag object which contains the native element and the object position in the viewport.
Is there any way to get the element's relative position to the boundary?

Hi! Thank you for your answer. I will check it soon and let you know.

Hi, did you have a chance to look on the issue?

Hi, did you have a chance to look on the issue?

Sorry for the late reply. It is justified by the fact of war in Ukraine. I didn't have enough time to process you request. 😥
But now I am ready to answer. 😉
After investigation I noticed that changes you have requested aren't necessary. It just adds redundant property to the ngxDragged event that can be calculated by yourself if needed.
Here is an example of retrieving element's relative position to the boundary with APIs available at the moment.
app.component.ts
import {Component, ElementRef, ViewChild} from '@angular/core';
import {NgxDrag} from 'ngx-drag-resize';

@Component({
  selector: 'app-root',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.scss']
})
export class AppComponent {
  /**
   * Draggable element's boundary
   *
   * @private
   */
  @ViewChild('dragArea') private readonly dragAreaRef: ElementRef<HTMLDivElement> | null = null;

  onDrag({top, right, bottom, left}: NgxDrag): void {
    const dragBoundaryClientRect = this.dragAreaRef?.nativeElement.getBoundingClientRect();

    if (!dragBoundaryClientRect) {
      return;
    }

    /**
     * Relative position of draggable element to its boundary
     */
    const position = {
      top: top - dragBoundaryClientRect.top,
      right: dragBoundaryClientRect.right - right,
      bottom: dragBoundaryClientRect.bottom - bottom,
      left: left - dragBoundaryClientRect.left
    };
  }
}

app.component.html
<section #dragArea>
  <div ngxDrag
       (ngxDragged)="onDrag($event)"
       [ngxDragBoundary]="dragArea"></div>
</section>

closed due to https://github.com/dmytro-parfenov/ngx-drag-resize/issues/4#issuecomment-1712836161


Do not duplicate symbols between libcrypto and libssl in static builds

Followup to #15704
This otherwise breaks linking applications to static libssl on some platforms.

@levitte Unfortunately the SHARED_SOURCE trick does not seem to work as expected.

That was easier to fix than I expected.

This pull request is ready to merge

Merged.


Let 365 fe image gallery on the project public page

This PR adds the project gallery on the project page header.
Testing instructions
testig link exemple: https://heco-invest-frontend-e6qs65qs7-vizzuality1.vercel.app/en/project/test-7-test-2
If the project has images, they will be shown on the top right on the project header.
If the user clicks an image, a modal will open with the images on a slider.
Tracking
LET-365

@barbara-chaves With which project have you tested this PR? I tested with /project/test-7-test but I'm unable to see any photo.

try with this:
https://heco-invest-frontend-e6qs65qs7-vizzuality1.vercel.app/en/project/test-7-test-2

Another issue I found, is that dragging the images/slides is bugged. T

I tested dragging the slides on chrome dev tools simulation of a touch screen and it works fine. It just doesn't work when clicking.


Can be flexible, for example, will only be displayed in a page?

RT!

Sorry, this feature is not planned at the moment.


Small refactor of util/autogen_md.py, bikeshed on the name

I'm trying to write some documentation for this, and was looking at the code so took the opportunity to do a small refactor, mostly adding docstrings.

However I do want to have a brief bikeshed on the name before it is more formally documented.
I think AUTOGEN is too-generic, we already have ip_autogen in the repo for ip that is rendered by topgen/ipGen, so I think we could have a more-specific name for this. Thoughts?
I would vote for DOCGEN, but are there any other ideas?
@HU90m @pamaury @jwnrt @timothytrippel @andreaskurth

That's a good point, DOCGEN sounds good

I agree DOCGEN is better than AUTOGEN

I think docgen is a good name and it's a good idea to avoid overloading the term autogen.

With the new commit it's harder to see the diff between autogen_md.py and docgen_md.py but when I looked at the diff previously it looked good to me.


Release heapster v0.18.2.

cc @piosz

Let me get the jenkins fix PR in first. Otherwise LGTM.

@mvdan: I posted a patch with this PR to fix jenkins.

Cool, thanks. LGTM

Thanks @vishh!


Decide on an OSS license

We need to decide on an OSS license to use for the eventual IaC that will reside in this repository.  This decision can also be used as a starting point / recommendation for future community projects.
Some starting thoughts:

AWS CDK uses the Apache 2.0 license
Good comparative chart: https://choosealicense.com/licenses/
According to this source Apache 2.0 is the most frequently used after MIT: https://www.whitesourcesoftware.com/resources/blog/open-source-licenses-trends-and-predictions/
Some additional info re: Apache 2.0 vs. MIT: https://www.quora.com/Whats-the-difference-between-Apache-v2-0-and-MIT-license-What-are-the-major-consequences-of-using-Apache-software-over-MIT

My "vote" is for Apache 2.0 license, if anyone else feels differently, please speak up :)

I normally just use MIT

I use MIT for my pet projects. Will read those links tomorrow.

A year is enough time to think about this. I'm going with MIT!


[work-in-progress] Add git-lfs support

Work in progress to support retrieving git-lfs backed artifacts with this resource.
See https://github.com/telia-oss/github-pr-resource/issues/163 for more context.

Tests are pending; I haven't actually tested this yet to see if it resolves the problem, even though the image builds successfully. Will update once I've tested.

I may not get to the testing of this until week after next, so please let me know if I should apply a label or something to this PR to keep it from popping up in y'all's queue before it's tested.

Okay, I got to do a little testing, and I determined just installing git-lfs isn't enough to get LFS artifacts fetched with the repository, at least not with my configuration.
Here's what my output looks like:
Initialized empty Git repository in /tmp/build/get/.git/

Switched to a new branch 'master'

Downloading blah.jar (7.6 MB)

Error downloading object: blah.jar (1b03b66): Smudge error: Error downloading blah.jar (1b03b66aed720bf4dd8b138e9753809239c03f6a31fc123b1a55c0fab0087b18): batch request: missing protocol: ""

This is happening because the standard git-lfs hooks are attempting to fetch the artifacts via a configured remote, but the repo doesn't have any remotes configured.
I'm going to update the label of this PR to indicate that it's a work-in-progress; next step is to figure out an alternative way to retrieve the LFS artifacts. Maybe I can retrieve the LFS artifact via its oid ref inside the pointer file via GitHub's API, or maybe it is okay to add remotes and allow for credentials to be passed in to do a fetch via https (I'd prefer not having to do that).

I ended up abandoning this work back in January, closing this PR


Restrict stylesheet only to root github.com domain

Currently the stylesheet applies to any github.com domain such as https://jobs.github.com/.

This can be restricted with a regexp.

Hi @extremecoders-re thanks for the suggestion. This fix will be pushed in the next release.

Apparently the one-click installer on userstyles made it apply to all github.com domains.
Otherwise there aint a problem if we manually specify to apply it to sites beginning with https://github.com/*

Issue fixed in v.1.40.
Thanks for the help.


MSID: 93698 Version: 2 DOI: 10.1101/2<IP_ADDRESS>8051

MSID: 93698
Version: 2
Preprint DOI: https://doi.org/10.1101/2<IP_ADDRESS>8051
Step 1. Awaiting reviews
Editorial to post reviews via hypothesis
Useful links:

DocMap: https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=93698
New model tracking: https://docs.google.com/spreadsheets/d/1_fHaoOy7hjyocptKtVJRijeNpUY4hBS7Ck_aVmx6ZJk/
Reviews on sciety: https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>8051

For trouble shooting (e.g. no Docmaps available):

DocMap issue addressing: https://miro.com/app/board/uXjVNCwK6EI=/
Explore DataHub DocMaps API: https://lookerstudio.google.com/reporting/4c2f0368-babb-4beb-b5b3-497e7e7b0f08/page/ejphD
Unmatched submissions and preprints: https://lookerstudio.google.com/u/0/reporting/9f86204f-3bf7-477c-9b18-5c5ef141bf69/page/p_gxi57ha93c
Unmatched manuscripts spreadsheet: https://docs.google.com/spreadsheets/d/15QcK8w-ssB7109RQEDtFpJPZ0J5HTGxoHa_2TtpMBbg/edit#gid=1336081641

Step 2. Preview reviewed preprint
Production QC content ahead of publication
Instructions:

QC preview: https://prod--epp.elifesciences.org/previews/93698v2
Update ticket with any problems (add blocked label)
When QC OK, add QC OK label to ticket and add publication date and time to https://docs.google.com/spreadsheets/d/1amAlKvdLcaDp5W8Z8g77NmkwbMF5n_u89ArSqPMO8jg
Move card to next column
(At end of the day post link in #enhanced-preprint and ask for PDF to be generated)

Useful links:

Preprint DOI :  https://doi.org/10.1101/2<IP_ADDRESS>8051
Confirm reviews returned by EPP: https://prod--epp.elifesciences.org/api/reviewed-preprints/93698/v2/reviews
To update the MECA path in the docmap: https://docs.google.com/spreadsheets/d/1mctCQuNFBjSn97Lihy7_vBO6z7-N-oqyLv4clyi6zHg

Step 3: Awaiting search reindex
This step adds the reviewed preprint to the homepage: https://elifesciences.org
The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.
Useful links:

Jenkins pipeline to reindex search can be triggered sooner or monitored here: https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 4: Published! PDF requested
Waiting for PDF to be generated
Useful links:

PDF tracking: https://docs.google.com/spreadsheets/d/106_XeDjmuBae7gexOTNzg60lapeqjl2aRn9DzupGyS8/

Step 5: Introduce PDF to data folder and git repo
Upload PDF to relevent folder in git repo https://github.com/elifesciences/enhanced-preprints-data/
Step 6: Done!

[ ] Kettle is on!

error: p element contains a junk character '□' which should be replaced or deleted.
error: p element contains a junk character '□' which should be replaced or deleted.


fixing typos, stale comments, broken link

The link to "HTTP protocol" in the intro doc is broken.
The other changes are updating out-of-date comments and error messages.

Can one of the admins verify this patch?

lgtm!


Add a required version plugin

GitHub Issue: https://github.com/Islandora/documentation/issues/1343
What does this Pull Request do?
Adds a plugin that enforces a minimum and maximum ansible version. The versions are loaded from a file ansible-version.yml in the root directory of the playbook.
How should this be tested?
I adjusted the ansible-version.yml to have either a minimum version greater than or a maximum version less than my current version and tried to run either

vagrant up
ansible-playbook -i inventory/vagrant -uvagrant -e'islandora_distro=ubuntu/bionic64' playbook.yml

If you run the vagrant up and the machine has already been built it will not fail and just start the machine, if the machine is not built then it will build the box and then fail.
For example, with ansible version 2.8.1
> ansible --version
ansible 2.8.1
  config file = /sw/var/www/DAM2/claw-playbook/ansible.cfg
  configured module search path = ['/Users/whikloj/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.8.1/libexec/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.5 (default, Nov  1 2019, 02:16:32) [Clang 11.0.0 (clang-1<IP_ADDRESS>)]

I changed ansible-version.yml to
---
  # Minimum version of ansible this playbook works with.
  islandora_ansible_version_min: 2.8.2
  # Maximum version of ansible this playbook works with.
  islandora_ansible_version_max: 2.8.7

Then
> ansible-playbook -i inventory/vagrant -uvagrant -e'islandora_distro=centos/7' playbook.yml
Islandora-Playbook restriction: only an Ansible version between 2.8.2 and 2.8.7 is supported.

or
> vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/bionic64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'ubuntu/bionic64' version '20190225.0.0' is up to date...
==> default: Setting the name of the VM: Islandora CLAW Ansible
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 8000 (guest) => 8000 (host) (adapter 1)
    default: 8080 (guest) => 8080 (host) (adapter 1)
    default: 3306 (guest) => 3306 (host) (adapter 1)

...

- Islandora-Devops.fits (1.0.0) is already installed, skipping.
- Islandora-Devops.grok (2.0.0) is already installed, skipping.
- Islandora-Devops.karaf (1.0.0) is already installed, skipping.
- Islandora-Devops.keymaster (1.0.0) is already installed, skipping.
- Islandora-Devops.matomo (1.0.0) is already installed, skipping.
    default: Running ansible-playbook...
Islandora-Playbook restriction: only an Ansible version between 2.8.2 and 2.8.7 is supported.
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Interested parties
@Islandora-Devops/committers @g7morris

Also @ruebot could test this.

This is great.  I'd also be interested in the version information for Ansible being on the README.md page as well, so users coming in can start by grabbing the right version and not finding out that they have the wrong one when they "vagrant up".

Based on the statement in the latest listserv post.

Ansible 2.5.1 installed (the Vagrant image fails when using the current version of Ansible (2.9.5), but sticking to 2.5.x seems to work).

https://groups.google.com/forum/#!msg/islandora/BlWgiu68zX8/SPVas7qFAAAJ
Maybe we can merge?

fwiw, I've used Ansible 2.8.7 (on a mac) successfully a number of times.  Just a data point, not any sort of comment on whether or not to merge.

So after sinking a nontrivial amount of time to get islandora-playbook to spin up with Vagrant, I can confirm that ansible v2.5.1 (the default version that you get with apt install ansible on Ubuntu 18.04) and v2.6.9 (the version that you get when you install ansible after adding the ppa:ansible/ansible repository) don't work, but ansible v2.8.10 does (requiring one to remove the ppa:ansible/ansible repo and then add the ppa:ansible/ansible-2.8 repo).
Having Vagrant error out due to my having the wrong version of ansible early on in the process would have saved me a lot of time by leading me down the proper path, so I'm adding my $0.02 that including this plugin to specify a minimum/maximum version is 100% A Good Thing For Users.


Webhooks for Appoinment Created / Updated / Deleted

Hi,
Is it possible to setup somethign like a webhook to trigger on new appoinments being Added / Updated / Deleted?
I'm looking to integrate Microsoft bookings with a lead automation system and will need that sort of functionality for it to work.
Thanks,
Nick

Thanks for the suggestion Nick. I'll forward it to the team.
In the meantime, it will be interesting to post this as an idea in the Microsoft Bookings Uservoice forum.
Another option would be to see if this can be already accomplished with the Graph Webhooks. I haven't tried it myself, but since Booking appointments are effectively stored as events, it is possible this already works.
For example, you should be able to access booking appointments as "normal" events with URLs like this: https://graph.microsoft.com/v1.0/users('businessId@yourcompany.com')/events
So I'm wondering if the Graph Webhook will work with these as well. Their doc talks about the /me/events, but not /users('...')/events, so I'm not 100% sure.

Any update on this feature?  This would be a great use-case for reporting.

Same here.

@fpintos can you confirm if it is possible to use the Graph Webhook ?
Thanks

@sven-borden @jfacoustic @LarsKemmann I haven't worked with the Bookings API in a while. I suggest using https://developer.microsoft.com/en-us/graph/support to submit questions and feature requests. Thanks.


Issue #33: Fix directed edges

This PR:

[x] Fix bug #33
[x] Adds 2 interfaces.
[x] Adds 1 trait.
[ ] Update code style.
[ ] Adds .editorconfig configuration file.
[ ] Adds .gitattributes file.
[ ] Use Grumphp for pre-commit checks.

Thanks for this PR,
You are introducing quite a lot of other things in this PR than just a fix for this issue you found. Please remove those. And make this a clean PR to fix the issue. Any additional changes can be discussed but should not be in this pr.
It is perfectly ok if you are using tools to improve your own code base, but I don't want them in this repo right now. If we want to introduce Grumphp for example I want to use it organisation wide and define how to introduce it. Not just in a single repo, without any additional that is organisation wide used.

Allrighty, removed all the unrelated stuff.

Waiting for your feedback @jaapio .

Is there any chance this is moving forward - and changes into master?


Manual installation documentation

Manual installation is based on docker containers.

Update README.md to add sample manual installation using docker
Change DROP KEYSPACE by DROP KEYSPACE IF EXISTS
Create Dockerfile for OSS Tracker console
Improve OSS Tracker scraper build.gradle

Related to #7 and should helps #6

@kakawait a quick read of this looks good.  I'm out of the office this week. Once I get back to the office I want to try this and then I'll merge.  Thanks!

@aspyker don't worry take your time :+1: In my case my instance is running well I'm not depending on this PR :)

Thanks for the PR!  Let's discuss the few items I brought up, but otherwise this is great.  I followed it and was able to run from Mac OSX docker-machine and it worked as expected.  I didn't, yet, validate the kibana part.

I fix few items @aspyker except cassandra performance

hello other windows users, has anyone encountered this error when executing docker run -d --name osstracker-cassandra -p ${CASS_PORT}:9042 cassandra command.
docker: An error occurred trying to connect: Post http://<IP_ADDRESS>:2375/v1.22/containers/create?name=osstracker-cassandra: dial tcp <IP_ADDRESS>:2375: connectex: target machine actively refused it..

maybe using our company's proxy solves this ... but not sure where to set the proxy?
if i can get osstracker and required components running locally, i promise to contribute the osstracker installation guide for windows ... :-)

Yeah error you quoted is docker client was not able to connect with docker daemon. Does docker machine correctly running ? Or do you cirrectly export var from docker-machine eval? Otherwise I may possible that corporate proxy intercept and deroute call to docker daemon. On Mac you need to add docker sock file on export no_proxy=.... I think there is something similar with Windows. But your problem is related to your docker setup.

I've given up on getting OSS Tracker running locally on a Windows--too many challenges getting Docker running (behind company VPN, under company security scans, etc.). I'm using a non-work, Mac computer--should have done this from the start because I got through in 5 minutes on a Mac what I couldn't get through over a period of days on a Window. But I'm getting not found error when trying to execute the cql:
bash-3.2$ docker run -it --link osstracker-cassandra:cassandra -v $(pwd)/osstracker-ddl/:/tmp/ddl/ --rm cassandra cqlsh cassandra -f /tmp/ddl/osstracker-simple.cql
Can't open '/tmp/ddl/osstracker-simple.cql': [Errno 2] No such file or directory: '/tmp/ddl/osstracker-simple.cql'
bash-3.2$

should this file be found in the cassandra image? i successfully pulled cassandra from library so presumably i should have the osstracker-simple.sql. any ideas?

I suppose you're using my fork because this PR is not merged so osstracker-simple.sql does not exist on upstream repo?
Otherwise on Windows (same on Mac) you have a VM in order to run docker (via docker-machine). So you must keep in mind when mounting (using -v) path, your are mounting path from your VM and not Windows  host.
But by default docker-machine create a Vbox share between Windows host and your VM. The mount is C:\Users. Does the source are inside this path (you can have any subfolder) ?
Else a fast workaround is too add manually these folder on VM using docker-machine scp (https://docs.docker.com/machine/reference/scp/) and for sure rewrite your docker run.
Example
docker-machine scp path/on/windows/osstracker-ddl/* default:/tmp/ddl
# you can check by connecting in SSH
docker SSH default
ls -l /tmp
exit
docker run -it --link osstracker-cassandra:cassandra -v /tmp/ddl:/tmp/ddl/ --rm cassandra cqlsh cassandra -f /tmp/ddl/osstracker-simple.cql

Be careful I didn't try it may missing some directroy creation or other simple command but the idea is here

@richnsoos now you can use official repo (branch documentation)
I know it can be hard to start with docker on non Linux env but I really think it would pay :) hard/standard installation is unmaintainable!
I could append some issue from your experience on documentation

@kakawait , thanks for hints; I will try them and report back!
The *.cql file from upstream (netflix/osstracker) is not osstracker-simple.cql, it's https://github.com/Netflix/osstracker/blob/master/osstracker-ddl/osstracker.cql. I will try this following command to init cassandra ddl and report back:
docker run -it --link osstracker-cassandra:cassandra -v /tmp/ddl:/tmp/ddl/ --rm cassandra cqlsh cassandra -f /tmp/ddl/osstracker.cql

congrats on getting your doc pr merged! i will contribute too where i can.

Yes on master branch but osstracker.cql uses NetworkTopologyStrategy replication that does not work with my documentation. You should really checkout documentation branch to retrieve osstracker-simple.cql which use SimpleStrategy replication
https://github.com/Netflix/osstracker/blob/documentation/osstracker-ddl/osstracker-simple.cql


Users wrong user

Hi!
My Dockerfile has this:
USER node

When I do docker exec -it foo-myapp-1 /bin/bash, I enter the container as the node user.
But when I do opener myapp, I enter the container as the root user.
How do I make opener enter as the non-root user? Preferrably as one-time configuration, since the whole point of opener is brevity.

Hello, thanks for your report!
You are right, default user to shell in to container is always root. I'll consider passing standard user option to connect command, it should be something like this:
opener myapp --user node

Not sure when exactly it'll be added, though.

Meanwhile workaround with su can be used if the container provides it
user:host~$ opener node
Found container with image `node` containing `node`
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
c1b87e1ceaad        node                "docker-entrypoint.s…"   11 minutes ago      Up 11 minutes                           serene_feynman

Found native shell: bash
Connecting...
root@c1b87e1ceaad:/# whoami
root
root@c1b87e1ceaad:/# su node
node@c1b87e1ceaad:/$ whoami
node
node@c1b87e1ceaad:/$

Should be su -s /bin/bash node in my case. It defeats the purpose of opener because together with i vocation of opener it's not shorter than docker exec -it my-cont-1 /bin/bash.

Being able to pass a user would be very useful. Actually, I do think that the default should be to use the user specified in the dockerfile, and to be able to pass --user root to log in as root if needed.


@rspack/plugin-html only providing simulated default export

System Info
System:
OS: Windows 10 10.0.19045
CPU: (16) x64 11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz
Memory: 17.15 GB / 31.73 GB
Binaries:
Node: 18.15.0 - C:\Program Files\nodejs\node.EXE
npm: 9.5.0 - C:\Program Files\nodejs\npm.CMD
Browsers:
Edge: Spartan (44.19041.1266.0), Chromium (112.0.1722.48)
Internet Explorer: 11.0.19041.1566
Details
Originally reported in #2795 but splitting into separate issue.
@rspack/plugin-html ships with a cjs build but that cjs build only provides a simulated default export.  That is, the exports object has a default property on it, but exports itself is not assigned to anything:
exports.default = HtmlRspackPlugin;

Using the are the types wrong tool, it will report:

The CJS module resolved at the package under contains a simulated export default with an __esModule marker, but no top-level module.exports. Node does not respect the __esModule marker, so accessing the intended default export will require a .default property access in Node from an ES module.

Effectively, this doesn't allow for an expected esm import or cjs require.  To use cjs one must:
const { default: HtmlRspackPlugin } = require('@rspack/plugin-html');

and to use esm one must:
import HtmlRspackPlugin from '@rspack/plugin-html';
...
	const plugins = [
		new HtmlRspackPlugin.default({
...

I'm guessing this syntax requirement is unintentional.
Reproduce link
No response
Reproduce Steps
Try using @rspack/plugin-import with either:
const HtmlRspackPlugin = require('@rspack/plugin-html');

or
import HtmlRspackPlugin from '@rspack/plugin-html');

If you try to new HtmlRspackPlugin() you will get an error about it not being constructable.

@ahabhgk cc

@shawnmcknight Do you want to create a PR for this?

@shawnmcknight Do you want to create a PR for this?

I can, but I'm not sure there is a way to get the tsc compiler to support a cjs "default" export (i.e. module.exports=HtmlRspackPlugin as well as allowing for cjs style "named" exports (i.e. module.exports.foo=Foo).
Currently the lib is exporting the HtmlRspackPlugin as an esm default export as well as exporting multiple named interfaces as well as defaultTemplateCompiler.  The easiest path forward would be to just export HtmlRspackPlugin as a named export but that would make the implementation different than the equivalent webpack plugin.
Alternatively, I believe I can make everything work with module.exports=HtmlRspackPlugin if the defaultTemplateCompiler doesn't need to be exported.  I'm not sure if there's a reason that's exported today.
I believe those are the only two viable approaches due to limitations in tsc's compilation output.  Any thoughts on which is preferable?

I think use named exports is the right choice but it's definitely a breaking change, we will fix this when release 0.2.0 version


CI: Make NAMD a neutral test for external forks

It does not work in the current state

This should hopefully work now (we can only really test it if someone else outside opens a new PR).

Seems like this works, tested with #3712


Inconsistent variable names in code examples

At first we use variable exp for the Experiment object. However, in the following code example there is a reference to variable named experiment which should be exp.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f70a36b8-6b3b-e06c-84d6-33fd56f79ea7
Version Independent ID: fd3dc619-fc56-5f93-860b-997c2eeb44f4
Content: Train scikit-learn machine learning models - Azure Machine Learning
Content Source: articles/machine-learning/how-to-train-scikit-learn.md
Service: machine-learning
Sub-service: core
GitHub Login: @maxluk
Microsoft Alias: maxluk

@DStekel3 we will review your feedback and get back to you shortly. Thanks.

@DStekel3 thanks for your feedback, will proceed to update the document accordingly. Thanks.


ESM<EMAIL_ADDRESS>breaks typescript path aliases

What happened?
Using the recommended way of instrumenting an ESM application causes path aliases to break, rendering it unusable.
Steps to Reproduce

Have a TS project that you start with tsx
Setup path aliases in your tsconfig & package.json
Use path aliases
Add<EMAIL_ADDRESS>to your startup command

Expected Result
It starts
Actual Result
It fails to start, erroring out on any imported files that use path aliases
Additional Details
OpenTelemetry Setup Code
/*instrumentation.ts*/
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

import { Resource } from '@opentelemetry/resources';
import { ConsoleMetricExporter, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

// NOTE: https://github.com/open-telemetry/opentelemetry-js/blob/main/doc/esm-support.md

const sdk = new NodeSDK({
    resource: new Resource({
        [ATTR_SERVICE_NAME]: 'yourServiceName',
        [ATTR_SERVICE_VERSION]: '1.0',
    }),
    metricReader: new PeriodicExportingMetricReader({
        exporter: new ConsoleMetricExporter(),
    }),
    instrumentations: [
        getNodeAutoInstrumentations({
            '@opentelemetry/instrumentation-http': {},
            '@opentelemetry/instrumentation-express': {},
        }),
    ],
});

sdk.start();

tsconfig.json
{
    "extends"<EMAIL_ADDRESS>    "include": [ "./**/*.ts"],
    "compilerOptions": {
        "outDir": "./dist",
        "rootDir": ".",
        "baseUrl": ".",
        "paths": {
            "@/*": ["./src/*"]
        },
        "strict": true
    }
}

{
    "$schema": "https://json.schemastore.org/tsconfig",
    "display": "Default",
    "exclude": ["node_modules"],
    "compilerOptions": {
        "lib": ["ESNext"],
        "target": "ESNext",
        "module": "ESNext",
        "moduleResolution": "Node",
        "outDir": "${configDir}/dist",
        "strict": true,
        "esModuleInterop": true,
        "forceConsistentCasingInFileNames": true,
        "skipLibCheck": true,
        "noEmitOnError": false,
        "experimentalDecorators": true,
        "resolveJsonModule": true
    }
}

package.json
{
    "name": "",
    "version": "1.47.3",
    "type": "module",
    "main": "src/infra/web/index.ts",
    "scripts": {
        "start:dev": "pnpm build:routes && dotenvx run --env-file=.env.development -- npx tsx<EMAIL_ADDRESS>--import ./telemetry.ts src/infra/web/index.ts",
        "build:routes": "tsoa -c tsoaConfig.json spec-and-routes",
    },
    "dependencies": {
        "@aws-sdk/client-connect": "catalog:aws-sdk",
        "@aws-sdk/client-dynamodb": "catalog:aws-sdk",
        "@aws-sdk/client-pinpoint": "catalog:aws-sdk",
        "@aws-sdk/client-s3": "catalog:aws-sdk",
        "@aws-sdk/client-ssm": "catalog:aws-sdk",
        "@aws-sdk/lib-dynamodb": "catalog:aws-sdk",
        "@aws-sdk/s3-request-presigner": "catalog:aws-sdk",
        "@aws-sdk/util-dynamodb": "catalog:aws-sdk",
        "@opentelemetry/sdk-node": "^0.54.0",
        "@opentelemetry/sdk-metrics": "^1.27.0",
        "@opentelemetry/instrumentation": "^0.54.0",
        "@opentelemetry/auto-instrumentations-node": "^0.52.0",
        "@opentelemetry/exporter-prometheus": "^0.54.0",
        "@opentelemetry/semantic-conventions": "^1.27.0",
        "@opentelemetry/resources": "^1.27.0",
        "@dotenvx/dotenvx": "^0.37.1",
        "@smithy/node-http-handler": "3.0.0",
        "@tsoa/runtime": "^6.2.1",
        "cors": "^2.8.5",
        "date-holidays": "^3.23.12",
        "dotenv": "*",
        "dynamodb-toolbox": "^0.9.5",
        "env-var": "^7.5.0",
        "express": "^4.19.2",
        "file-type": "^19.1.0",
        "fluent-ffmpeg": "^2.1.3",
        "jsonwebtoken": "^9.0.2",
        "ky": "^1.3.0",
        "libphonenumber-js": "^1.11.4",
        "lodash-es": "^4.17.21",
        "luxon": "^3.5.0",
        "module-alias": "^2.2.3",
        "tsoa": "^6.2.1",
        "type-fest": "^4.18.3",
        "typia": "^6.10.4",
        "uuid": "*"
    },
    "devDependencies": {
        "@types/jsonwebtoken": "^9.0.6",
        "@faker-js/faker": "^8.4.1",
        "@hyrious/esbuild-plugin-commonjs": "^0.2.4",
        "@types/chai": "^4.3.19",
        "@types/cors": "^2.8.17",
        "@types/express": "^4.17.21",
        "@types/fluent-ffmpeg": "^2.1.24",
        "@types/lodash-es": "^4.17.12",
        "@types/luxon": "^3.4.2",
        "@types/module-alias": "^2.0.4",
        "@types/supertest": "^6.0.2",
        "@types/uuid": "*",
        "aws-sdk-client-mock": "^4.0.1",
        "copyfiles": "^2.4.1",
        "deepmerge": "^4.3.1",
        "supertest": "^7.0.0",
        "ts-add-js-extension": "^1.6.4",
        "ts-patch": "^3.2.1",
        "ts-runtime-checks": "^0.6.2",
        "tsc-alias": "^1.8.10",
        "tsx": "^4.11.2",
        "typescript": "^5.6.2",
        "vitest-tsconfig-paths": "^3.4.1"
    },
    "_moduleAliases": {
        "@": "src"
    }
}

Relevant log output
No response

.

hey @douglasg14b would you be able to provide the tsconfig.json you're using as well? Thanks

@jpmunz tsconfig added to OP

@douglasg14b I believe this behaviour is coming from the import-in-the-middle package used by @opentelemetry/instrumentation/hook.mjs. I setup a minimal example here to reproduce without opentelemetry: https://github.com/jpmunz/simple-node-app
I'm not actually sure if this package is intended to understand those path aliases, I'm asking about that here https://github.com/nodejs/import-in-the-middle/issues/162, if not there would need to be some additional step in your build process to resolve those paths prior to running the loader

Hm, I'm not entirely sure how paths would be resolved before using tsx since tsc-alias is expected to run after compilation. Which happens during tsx run.
Precompiling would defeat the purpose of using tsx (For which there are several reasons)

so to get that working with tsx I think the hook would have to be wrapped with something that could resolve the path aliases at run time. You might be able to give this workaround (based on https://stackoverflow.com/a/75318315) a try, it uses tsconfig-paths to resolve the aliases before passing them to the opentelemetry hook:
// wrapper_hook.mjs
import {
  load,
  resolve as origResolve,
  getFormat,
  getSource,
} from<EMAIL_ADDRESS>
// See https://stackoverflow.com/a/75318315
import {loadConfig, createMatchPath} from 'tsconfig-paths';
import {pathToFileURL} from 'url'
const {absoluteBaseUrl, paths} = loadConfig()
const matchPath = createMatchPath(absoluteBaseUrl, paths)
const wrappedResolve = (specifier, ctx, defaultResolve) => {
  const match = matchPath(specifier, undefined, undefined, [".ts", ".tsx"]);
  const updatedSpecifier = match
    ? pathToFileURL(match).href : specifier;

  return origResolve(updatedSpecifier, ctx, defaultResolve);
}

export { load, wrappedResolve as resolve, getFormat, getSource };

and then use that wrapper as the loader instead of<EMAIL_ADDRESS>npx tsx --experimental-loader=./wrapper_hook.mjs --import ./telemetry.mjs <your-script>.ts

I put that together in an example here where I use path aliases and was able to see spans for http requests from the auto instrumentation working: https://github.com/jpmunz/simple-node-app/tree/otel-loading-example
(Note: I saw in your example that the setup file was telemetry.ts but I was getting import-in-the-middle complaining about that so had to rename to telemetry.mjs)

I will try this soon, but I do want to note that I get http metrics regardless of using the experimental-loader script.
This appears to avoid a crashing error, however, I don't get anymore than http metrics, which I receive the same (And in the same detail) without this. Express does not appear to be instrumented this way.

Is this expected behavior? That http is instrumented (regardless of the in-the-middle usage). But express is not either way?

@jpmunz Using the method described in the linked issue resolve the problem for me.

I no longer need to use --experimental-loader=./optWrapperHook.mjs
I can use telemetry.ts now without issue
http and express are now instrumented correctly

nice! If that worked it seems like the way to make this easier going forward, as that other issues suggests, is with module.register as opposed to the experiment loader stuff that is currently recommended

Is this expected behavior? That http is instrumented (regardless of the in-the-middle usage). But express is not either way? If not, is there configuration changes I should be trying?

@douglasg14b for this I'd recommended filing another issue, if you aren't getting telemetry from express either way that seems separate from the original problem outlined here around path aliases


Updated ckeditor to version 4.5.9

I updated the ckeditor version to support some new plugins from the official site, but I don't know if this will break other dev's build if they decided to update blogo.
Maybe merge this as a new version of blogo?

Coverage remained the same at 98.751% when pulling b7d139d03a98519c28c88460764014df2434d020 on SuhanKoh:master into eee0a8854cfdc309763197e3ef9b295008be85f6 on greyblake:master.

Coverage decreased (-0.09%) to 98.663% when pulling c7b5ba7ed7a9d99454cabe3f06208a54dad627c3 on SuhanKoh:master into eee0a8854cfdc309763197e3ef9b295008be85f6 on greyblake:master.

Archives feature wasn't intended in the pr.


uv build failing to build wheel correctly when project.name != package folder name

I have a project with a name (i.e. foo) that's not the same as the source folder / package name (src/bar).
[project]
name = "foo"
version = "1.2.3"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/bar"]

[tool.hatch.build.targets.sdist]
packages = ["src/bar"]

If I run uvx hatch build I get a wheel and sdist that contain the files I expect - i.e. the wheel {project.name}-{version}-py3-none-any.whl  contains
{project.name}-{version}.distinfo/
    METADATA
    RECORD
    WHEEL
bar/
     {contents of src/bar}

But if I run uv build the sdist is correct, but the wheel only contains the project metadata - i.e. the contents of the archive {project.name}-{version}-py3-none-any.whl is
{project.name}-{version}.distinfo/
    METADATA
    RECORD
    WHEEL

i.e. bar/ is missing
It works if I set the project name the same as the packages folder. I would expect that uv build and uvx hatch build would behave exactly the same?
I'm using uv v0.4.24

Could you create the sdist and check if it has the contents you expect. I think it's missing files because of your sdist config.

The sdist  works fine, the wheel is built correctly when I use hatch but not uv with the same pyproject file. Both have the same configSent from my iPhoneOn 21 Oct 2024, at 6:37 pm, bluss @.***> wrote:
Could you create the sdist and check if it has the contents you expect. I think it's missing files because of your sdist config.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

Could this be another case of https://github.com/astral-sh/uv/pull/8220?

I assume uv build --wheel and uv build --sdist produce the right artifacts? (That's not the same as running uv build, which builds the wheel from the sdist.)

@charliermarsh yes it does

Does python -m build work on your project, or does it produce similar builds? (https://github.com/pypa/build)

@charliermarsh no that does the same thing as uv build and only includes the metadata in the wheel, but the source is included in the sdist

@david-waterworth -- You may have a real issue with your project then. pypa/build is effectively the reference implementation for the spec. My guess is that uvx hatch build builds the wheel and the sdist from source, rather than building the wheel from the sdist as is the default in pypa/build and uv. You'll either need to figure out why your sdist isn't working as expected or use uv build --sdist --wheel (but, again, your sdist may have a real problem?).

foo.zip
I created a minimal example - maybe there is something wrong with the sdist, I've not actually tried to install it. Also https://hatch.pypa.io/latest/config/build/#packages doesn't appear to explicitly cover the case where project.name doesn't equal package name - maybe it's not supported, the accepted answer to https://stackoverflow.com/questions/77070305/how-to-distribute-a-python-package-where-import-name-is-different-than-project-n was to switch to setuptools?

Adding
[tool.hatch.build.targets.wheel]
packages = ["src/bar"]
include = [
  "src/bar/**",
]

Seems to work

I'm going to close for now as I don't think it's uv-specific.


Cannot find name 'CheerioStatic'.

NodeJS app. npm install --save truncate-html
Works fine when working locally and in development. But as soon as I do "npm run build" it fails.
node_modules/truncate-html/dist/truncate.d.ts:52:21 - error TS2304: Cannot find name 'CheerioStatic'.

52     (html: string | CheerioStatic, length?: number, options?: IOptions): string;
                       ~~~~~~~~~~~~~

node_modules/truncate-html/dist/truncate.d.ts:53:21 - error TS2304: Cannot find name 'CheerioStatic'.

53     (html: string | CheerioStatic, options?: IOptions): string;
                       ~~~~~~~~~~~~~

just fixed this issue, please update to the latest version v1.0.4 and try again


Fixing up Windows TestFileManager

_copyOrLinkDirectoryHelper should throw on errors
Accounting for that Windows permissions aren't compatible with posix ones
When given "" as a file name, most Windows apis return file not found, not invalid filename

cc @compnerd
@swift-ci please test linux

Removed the NSError changes, and have Windows follow posix such that "" is an invalid file instead of being an unfound file.
@swift-ci please test

@swift-ci please test

Rebase
@swift-ci please test linux

@swift-ci please test linux

I think that doing the remapping in the _NSErrorWithWindowsErrno is better.  The new function is less likely to get used I think.

I think that doing the remapping in the _NSErrorWithWindowsErrno is better. The new function is less likely to get used I think.

Fair, I've added a [Path]? parameter to _NSErrorWithWindowsErrno instead

@swift-ci please test linux

Thanks, this LGTM.


Remove extraneous question mark in section heading

The heading "How editable installations work?" either contains a typo (the question mark should be omitted), or else the heading as written is ungrammatical because it is missing a word ("How do editable installations work?").

Summary of changes
Other headings in this section are not written as questions, so this commit removes the question mark, based on the assumption that the question mark is extraneous.
Pull Request Checklist

[ ] Changes have tests. Not relevant.
[ ] News fragment added in [changelog.d/].
(See [documentation][PR docs] for details)

This is such a small typo that it does not seem to warrant a changelog.d file.

Thank you very much @mhucka


Bugfix: drop context and stop caching TrackedAsyncData

We had a race condition with the existing waiter teardown, because of an additional, unnecessary, teardown we had in the destructor.

Considering this also highlighted that we did not need the cache of Promises to TrackedAsyncData. The cache had some non-trivial overhead, and it also risked leaking if anything held onto a strong reference to the Promise.

This is breaking, technically, but is a high-urgency fix so we are treating it as a bugfix.

@SergeAstapov I updated, mind taking another gander?

@chriskrycho looks like two more places in readme missing:
return new TrackedAsyncData(this.store.findRecord('user', userId), this);
return load(this.store.findRecord('user', this.args.id), this);

otherwise looks good!

Ah, good! Fixed, thanks. Will enable auto-merge.

Done and out as 1.0.1. Thanks again for the speedy reviews @SergeAstapov!


Unable to get recorded file content (read recorded file as text)

Hello, sorry for the issue, but i can't understand who to record and read an audio file, i am using this code:
I first created a www/audio.wav empty file, then:
_ .run(function($ionicPlatform, $cordovaMedia, $cordovaFile) {
  $ionicPlatform.ready(function() {
var src = "audio.wav";

var media = $cordovaMedia.newMedia(src);
media.startRecord();

// success
console.log('Srtarted recording.....');

$cordovaFile.checkFile("documents://audio.wav").then(function(result) {
  // Success!
  console.log('checking file ....');
  console.log(result);
}, function(err) {
  // An error occurred. Show a message to the user

  console.log('ERROR CHECKING FILE');
  console.log(err);
});

});
});
It doesn't work, no errors, and no console.logs in xcode console, i already added all the console,media,file cordova plugins to the platform.
I just would like to record an audio.wav file and get its content as text, what i am doing wrong?
I am using ios emulator for this
thank you

I am not even able to use the core $cordovaFile functions.
Can u help
Thanks in advance
<EMAIL_ADDRESS>


Improve documentation regarding folder logic

With https://github.com/SAP2Moose/SAP2Moose/issues/78 it was proposed to change the folder logic of SAP2Moose to PREFIX. This was possible but is no trivial change. Small problems occured. Whenever such a change is done in the futures, it is to be expected, that problems occur and issues are send to the abapGit contributors. To prevent this workload, the documentation should be improved.
Please check my proposal, did I understand the arguments correctly?

Looks good, however it is not possible to install anything to $TMP in abapGit, https://github.com/larshp/abapGit/blob/c5158bb3330a75141e8ab7bf3be852dcc32ecaa2/src/zabapgit_repo_impl.prog.abap#L960

So you will change my formulation regarding $TMP? You meant really $?

updated, merging, thanks 😄


[Bug] Call hadoop job list in php function，process has exited,exitStatusCode:0,but hadoop job list Been running

Call hadoop job list in php function，process has exited,exitStatusCode:0,but hadoop job list Been running, worker Been running。
也就是：
php主方法run，调用了get_job_list（hadoop job -list）。
[taskAppId=TASK-37-989222-1008351]:[202] - process has exited, execute path:/home/shared/opt/soft/ds/tmp/dolphinscheduler/exec/process/12/37/989222/1008351, processId:30574 ,exitStatusCode:0 正常退出，
但是：后面还在执行php中的方法中的命令：$cmd = "hadoop job -list ";
[INFO] 2021-02-23 16:35:30.780  - [taskAppId=TASK-37-989222-1008351]:[415] - find app id: application_1613943087944_61608
[INFO] 2021-02-23 16:35:30.780  - [taskAppId=TASK-37-989222-1008351]:[415] - find app id: application_1613943087944_61791
导致worker进程一直在running。

Which version? Because the Hadoop task is still running in YARN, all the task pages show that it is running.

1.3.1。

hadoop job -list 是php主方法调用的函数里面的命令。

Look at the 'find app id: application_1613943087944_61791' in your log,It should be your task that starts a yarn task and then starts the PHP monitoring script. The current task display is actually running because the previous yarn task is running

看你的日志上记录‘ find app id: application_1613943087944_61791’ ，应该是你这个任务启动了一个yarn任务，之后启动了php 的监控脚本。目前任务显示真正运行就是因为前面的yarn任务是正在运行状态

hadoop job -list

hadoop job -list
you'd better not to use this command.

this command will list all the running yarn tasks to the task instance log, these yarn tasks will generate many application ids, DolphinScheduler will monitor all these yarn task until to finish, this operation will cost too  much time

完美解决！


SOF-1235 Change Skema mapping in Blueprint config

The old Skema is replaced by the new GFX Skema Templates. It should be noted that the acceptance criteria:
"If an entry has both an iNews name and iNews column, the iNews column takes priority." has not been considered, since this isn't relevant or possible to do until further implementation later in the phase regarding Overlay Mapping.
Maybe we should have a conversation about the naming of config tables as well. Especially how "Graphics Setups" should probably be "Graphic Setups", and that we use "Skema" in the user facing strings rather than the english name.

Changed target branch from develop to staging so it can be included in release 46.1.


fix: bump deps

:tada: This PR is included in version 1.0.7 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


How to work on this

Hi,
I am new to this CMS I don't know much about this CMS.
I am using win7 32 bit
Currently My task to create simple cms for my static sites which we are going to convert in angular-4.
I decided to use this cms 3 days completed neither start admin nor front...
Please guide me on this
INITDB=1 nps dev
nps executing: nps -p dev.admin,dev.api
nps executing: cd admin && nps dev
nps executing: cd api && nps dev
nps executing: ng serve --port 9000 --hmr true -e hmr --host <IP_ADDRESS> --progress               false --proxy-config proxy.conf.json
nps executing: NODE_ENV=development nodemon server/server.js --watch common --wa              tch server --ext js,json
[nodemon] 1.11.0
[nodemon] to restart at any time, enter rs
[nodemon] watching: D:\node-projects\colmena-cms\api\common//* D:\node-project              s\colmena-cms\api\server//*
[nodemon] starting node server/server.js
User: registration enabled
Container does not exist, creating container default
internal/streams/legacy.js:59
throw er; // Unhandled stream error in pipe.
^
Error: getaddrinfo ENOTFOUND images.freeimages.com images.freeimages.com:80
at errnoException (dns.js:28:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:73:26)
[nodemon] app crashed - waiting for file changes before starting...
NOTICE Hot Module Replacement (HMR) is enabled for the dev server.
The project will still live reload when HMR is enabled,
but to take advantage of HMR additional application code is required
(not included in an Angular CLI project by default).
See https://webpack.github.io/docs/hot-module-replacement.html
for information on working with HMR for Webpack.
** NG Live Development Server is running on http://<IP_ADDRESS>:9000 **
[HPM] Proxy created: /config.js  ->  http://<IP_ADDRESS>:3000/config.js
[HPM] Subscribed to http-proxy events:  [ 'error', 'close' ]
image

Hi I have make some changes & it's work for me now
System Windows 7 32Bit
Please Follow below URL
https://github.com/colmena/colmena-cms
Colmena CMS Change to makes it Work
npm install -g win-node-env
colmena-cms\admin\protractor.conf.js
baseUrl: 'http://<IP_ADDRESS>:9000/', >> baseUrl: 'http://localhost:9000/',
colmena-cms\api\config\default.yaml
baseUrl: http://<IP_ADDRESS>:3000 >> baseUrl: http://localhost:3000
url: http://<IP_ADDRESS>:9000 >> url: http://localhost:9000
colmena-cms\admin\src\lib\lb-sdk\lb.config.ts
private static path: string = '//<IP_ADDRESS>:3000'; >> private static path: string = '//localhost:3000';
colmena-cms\admin\proxy.conf.json
"target": "http://<IP_ADDRESS>:3000/config.js", >> "target": "http://localhost:3000/config.js",
colmena-cms\api\server\config.json
"host": "<IP_ADDRESS>", >> "host": "localhost",

@chetanbane the project should now be better usable on Windows.
Please try the latest develop branch. If it does not work, feel free to open a new issue and I'll make sure it works on Windows :)