Add some textual context to images, videos and audio in result.media, so that it can be useful in RAG

Issue
Currently, all images, video and audio in the crawled page are returned in result.media like follows.
{
audio:[],
images:[
{
src:"https://cdn.com/path/to/image",
alt:"mobile_integrations_560px",
type:"image"
},
.
.
.
],
videos:[]
}

Each entry in this object list contains a link/url from where the asset could be downloaded, the text passed in the alt attribute of the corresponding <img> HTML tag and a type field(since the entries are already grouped by audio, video and images not sure why we need the type field again)
Other than this there's no useful information regarding what may be the content of the media file (It could be architecture diagram, testimonial, partners, integrations etc).
So we have to entirely rely on the alt attribute to vectorise and store against it the link to the media file so that it can be retrieved during RAG. Also theres many times where I found utterly garbage if not misleading values in alt attribute of image tags of even well known companies.
I'm talking about a use case like perplexity (but for a company internal knowledge base etc)

Suggestion
I suggest that we add some textual context based on the position ie where the image is found with in the page. Perhaps the closest paragraph or header (right above the image). @unclecode please let me know your thoughts on this or if there's a better way to do RAG on images, for user queries.

Very interesting point that you mentioned, and it's actually been on my mind. Initially, I started by just extracting the media tags to have them ready. I have two strategies in mind.
First, I'm looking for a very small image-to-text model to annotate images and create descriptions. I've found a few or am working on one myself. This model needs to be efficient, not large, as we only need small captions and want to run it in parallel without heavy GPU dependence. There are many media sources on a page and need to be fast, otherwise it will be time consuming.
Second, there's a statistical NLP approach uses TF-IDF to gather information around the image, pulling in relevant text from the website, maybe even from other parts that refer to the image initially. This doesn't need a model, just a statistical method.
Since you brought this up, I'll start with the statistical approach while continuing my research and development on the LLM. Thank you for bringing this to my attention.

@unclecode Sounds good! Please let me know if can be of any help in research or coding. Since this library is slowly turning into a critical piece in one of my projects, I'm more than happy to help.

@aravindkarnam  I really appreciate your support and enthusiasm! You are most welcome to help with this enhancement. I'd like to focus on statistical and lexical-level NLP approaches initially. If these work well, we'll integrate them into the current codebase and update the documentation accordingly. While I'm keeping an eye on lightweight image-to-text models for generating captions, I'd like to put that on hold for now to see how our initial approach pans out.
For our immediate next steps, I propose the following experiment:

Develop a function that, for each image element:
a. Identifies the closest parent element.
b. Extracts the text surrounding the image.
c. If the extracted text doesn't meet a certain word count threshold, moves to the next parent element.
d. Repeats this process until we have sufficient contextual text.

Apply this function to extract text and corresponding images from 50 different websites, creating a diverse dataset.

Write a script that uses a multimodal model (such as Claude-3,5 Sonnet, GPT-4o, LLAVA, or Moondream) to evaluate our extracted text. The script should:
a. Pass both the image and the extracted text to the model.
b. Ask the model to evaluate if the text is a good representative description of the image (return o or 1)
c. Collect the model's assessment for each image-text pair, and finally calculate the accuracy.

Analyze the results to determine the accuracy of our approach.

If the results are promising, we'll use this method as a basis for caption generation and apply TF-IDF to extract keywords, making the information more interconnected with the rest of the page.

If the results aren't satisfactory, we'll analyze the problematic samples, gather insights, and refine our strategy accordingly.

So, would you be interested in taking on this experiment? If so, which parts would you like to work on? If you're open to it, you can handle the entire research to determine if this approach works well or not. Let me know. I'm happy to provide any necessary guidance or collaborate on specific aspects.
Let me know your thoughts on this approach and if you have any suggestions or modifications. I'm excited to see where this leads us!

@unclecode Sure. All the steps make sense. I'll take on this experiment. steps 1&2 sound straight forward. For 3&4 I haven't used a multimodal model in an API before, so let me give that a try and figure it out.
for 5: I conceptually understand TF-IDF but never really used it before. So I may need some help with that. Anyway let me start my work on this I'll update progress or blockers here.

@aravindkarnam Fantastic, glad to have you onboard! No worries about 3 and 5. I'll share a few code snippets from our library to speed things up. To evaluate your extracted text, use the following code:
import anthropic
import json
import base64
from typing import Dict, Any

def get_evaluation_prompt(extracted_description: str) -> str:
    return f"""
    Analyze the given image and the provided description. The description was generated by an automated system.

    Description: {extracted_description}

    Please evaluate the quality and relevance of this description in relation to the image.
    Return your evaluation as a JSON object with the following structure:
    {{
        "evaluation": string,  // One of: "relevant", "not_clear", "not_relevant"
        "score": number,  // A score from 0 to 100
        "explanation": string  // A short and concise explanation of your evaluation
    }}

    Ensure your response is wrapped in an XML tag called <result>. Make sure your JSON is valu and parseable.
    """

def evaluate_image_description(image_path: str, extracted_description: str) -> Dict[str, Any]:
    try:
        client = anthropic.Anthropic()

        with open(image_path, "rb") as image_file:
            image_data = base64.b64encode(image_file.read()).decode("utf-8")

        media_type = "image/jpeg" if image_path.lower().endswith(('.jpg', '.jpeg')) else "image/png"

        prompt = get_evaluation_prompt(extracted_description)

        response = client.messages.create(
            model="claude-3-5-sonnet-20240620",
            max_tokens=1024,
            messages=[
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "image",
                            "source": {
                                "type": "base64",
                                "media_type": media_type,
                                "data": image_data,
                            },
                        },
                        {
                            "type": "text",
                            "text": prompt
                        }
                    ],
                }
            ],
        )

        result_xml = " ".join([c.text for c in response.content if c.type == 'text'])

        # Extract content within <result> tags
        start_tag = "<result>"
        end_tag = "</result>"
        start_index = result_xml.find(start_tag)
        end_index = result_xml.find(end_tag)

        if start_index != -1 and end_index != -1:
            json_str = result_xml[start_index + len(start_tag):end_index].strip()
            result = json.loads(json_str)
            return result
        else:
            return {"error": "Could not find result tags in the response"}

    except FileNotFoundError:
        return {"error": "Image file not found"}
    except anthropic.APIError as e:
        return {"error": f"API error: {str(e)}"}
    except json.JSONDecodeError:
        return {"error": "Failed to parse JSON response"}
    except Exception as e:
        return {"error": f"Unexpected error: {str(e)}"}

# Example usage
result = evaluate_image_description("bird.jpeg", "This is a bird and tree.")
print(result)

You can keep experimenting and see how you can play around to get more often "relevant" results. You can also use the GPT-4 API. For TF-IDF, although we'll focus on it later, here is a sample code. You pass a query and a long text (in our case, the extracted markdown). It breaks the text into paragraphs and includes a third parameter to ensure we have rich paragraphs, especially in extracted markdown where some paragraphs might only have a few words. This will give better quality results.
from typing import List, Tuple
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

def chunk_paragraphs(paragraphs: List[str], min_word_count: int) -> List[str]:
    """
    Merge paragraphs sequentially to ensure each chunk has at least the minimum word count.

    Args:
    paragraphs (List[str]): List of original paragraphs.
    min_word_count (int): Minimum number of words for each chunk.

    Returns:
    List[str]: List of paragraph chunks.
    """
    chunks = []
    current_chunk = []
    current_word_count = 0

    for paragraph in paragraphs:
        words = paragraph.split()
        if current_word_count + len(words) < min_word_count and current_chunk:
            current_chunk.append(paragraph)
            current_word_count += len(words)
        else:
            if current_chunk:
                chunks.append(' '.join(current_chunk))
            current_chunk = [paragraph]
            current_word_count = len(words)

    if current_chunk:
        chunks.append(' '.join(current_chunk))

    return chunks

def sort_paragraphs_by_relevance(query: str, text: str, min_word_count: int = 50) -> List[Tuple[str, float]]:
    """
    Break text into paragraphs, chunk them based on minimum word count,
    calculate their relevance to the query using TF-IDF, and return chunks sorted by relevance.

    Args:
    query (str): The search query.
    text (str): The long text to be analyzed.
    min_word_count (int): Minimum number of words for each paragraph chunk.

    Returns:
    List[Tuple[str, float]]: A list of tuples containing paragraph chunks and their relevance scores,
                             sorted in descending order of relevance.
    """
    # Split text into paragraphs
    paragraphs = [p.strip() for p in text.split('\n\n') if p.strip()]

    # Chunk paragraphs based on minimum word count
    paragraph_chunks = chunk_paragraphs(paragraphs, min_word_count)

    # Add query to the list of documents for vectorization
    documents = [query] + paragraph_chunks

    # Create TF-IDF vectorizer and transform documents
    vectorizer = TfidfVectorizer()
    tfidf_matrix = vectorizer.fit_transform(documents)

    # Calculate cosine similarity between query and each paragraph chunk
    query_vector = tfidf_matrix[0]
    chunk_vectors = tfidf_matrix[1:]
    similarities = cosine_similarity(query_vector, chunk_vectors).flatten()

    # Create list of (chunk, similarity) tuples
    chunk_similarities = list(zip(paragraph_chunks, similarities))

    # Sort chunks by similarity in descending order
    sorted_chunks = sorted(chunk_similarities, key=lambda x: x[1], reverse=True)

    return sorted_chunks

# Example usage
if __name__ == "__main__":
    sample_text = """
    Python is a high-level, interpreted programming language.
    It was created by Guido van Rossum and first released in 1991.

    Python's design philosophy emphasizes code readability with its notable use of significant whitespace.
    Its language constructs and object-oriented approach aim to help programmers write clear, logical code for small and large-scale projects.

    Python is dynamically typed and garbage-collected.
    It supports multiple programming paradigms, including structured (particularly, procedural), object-oriented, and functional programming.

    Python is often described as a "batteries included" language due to its comprehensive standard library.
    It has a large and comprehensive standard library, which is one of Python's greatest strengths.

    Python's syntax allows programmers to express concepts in fewer lines of code than would be possible in languages such as C++ or Java.
    The language provides constructs intended to enable clear programs on both a small and large scale.
    """

    query = "Python programming features"
    min_word_count = 30

    results = sort_paragraphs_by_relevance(query, sample_text, min_word_count)

    print(f"Paragraph chunks sorted by relevance to the query (min {min_word_count} words):")
    for chunk, score in results:
        print(f"\nRelevance score: {score:.4f}")
        print(f"Word count: {len(chunk.split())}")
        print(chunk)

Then you can start and keep me updated on the progress. I think this will be a great and useful feature.

@unclecode I've implemented step 1 and after looking at results had a couple of thoughts.

Not all images scraped from a webpage are equal. For example there are hero images, section images etc such as this, which contain useful information for RAG while others are merely icons on buttons and links. For example a button called "Take me there" may contain a right arrow icon, such as this. There's no value in either storing this image or getting this image transcribed with a model. The sheer number of these images I'm getting from crawling even very simple pages is quite high.
We should implement a very simple filtration strategy to weed out such useless images at very early stage. This will save some cost downstream as well when we are passing images to models. Here are some proposals I have on filtration:
a) Create a cut-off for images based on height & width. For eg: eliminate images that are less than 100px in either height or width. We can expose this as a param to the user with a default value (similar to the word_count_threshold param)
b) Eliminate based on parent elements. From what I've seen mostly such small images are part of elements like buttons and links that open up other pages/navigation elements.

If we use a combination of a & b we can clear a lot of clutter upfront. Your thoughts?

Text from nearest parent looks quite useful so far. I've only inspected it manually with 3-4 sites, I'll evaluate with your script shortly.

@aravindkarnam  You're absolutely spot on about the image filtering! It's super common to have loads of duplicate images across pages. So here's what I'm thinking: Let's create a scoring system for each image and chuck out anything below a certain threshold. Here's a quick rundown of what we could score on (for each item we score 1):

Size: Bigger usually means more important. (score 1 if width and height > 100)
File size: Heftier files tend to be meatier content. (score 1 if size > 10,000 bytes)
Filename/path: Avoid stuff with "icon" or "button" in the name.
HTML context: Skip images in buttons or with "icon" classes.
Alt text: Meaningful alt text usually means it's a content image.
Image format: Prioritize JPEGs and PNGs over SVGs and GIFs.
Position: Images higher up the page often matter more. (Look at the index of the image element and check against the total length of images; if their rate is above 0.5, score it)
Duplication: Keep only one copy of each unique image. (Create a simple hash by mixing width and height like f"{width}x{height}:{src}", then use that to avoid duplication—this is not the best way, but it's fast)

The key here is speed – we're dealing with potentially thousands of images, so every millisecond counts. For example, to get image sizes, use a HEAD request instead of GET to the image URL. It's way faster because you're not downloading the whole image, just the metadata ('Content-Length').
Wrap up all these and make this score function in the way it gets the "soup" object, base_url, and min_score (I feel 4 is good) and returns a filtered list of image elements. Then plug that into our main crawling function.
Let me know what you think or if you need any clarification. Excited to see what you come up with!

@unclecode Completed the scoring function and now the results are much more cleaner. Proceeding to step 3, ie getting the results evaluated with a model. Aligned on all factors for scoring except 6 and 8.

Image format: Prioritize JPEGs and PNGs over SVGs and GIFs.

SVGs are preferred over JPG and PNG by developers because it offers better resolution independence and animation. In fact off late lottiefiles are very popluar and used in landing pages which use SVG or canvas to render images, we should even plan to get those SVG images converted to a JPG or GIF and caption them in future. For example checkout these pages the hero image is an animated SVG (urbanpiper.com, memberstack.com)

f"{width}x{height}:{src}"
Why don't we just do src, why do we have to prepend width and height. Already the image stored in src is just going to be the same file regardless. So we can just get the image captioned and stored.

Here are next steps that I'll plan

[ ] Get a good mix of 50+ sites, ranging from product landing pages, wikipedia, E-commerce, Documentation, knowledgebase etc. Filter the images and get the text from nearest parent and load it in a pandas DF.
[ ] Get the text rated by Multimodal AI model
[ ] Export the dataframe and upload here.

Then we can plan to tweak it further to improve the scores.
Have been a little busy yesterday and today. I'll complete this in next 2-3 days

@aravindkarnam, Congratulations on completing steps 1 and 2 and moving on to step 3. Let me address some of your questions.
Regarding image formats, it's somewhat subjective. For programmers, SVGs are great due to their vector-based nature, allowing us to use and render them in any size easily. However, our goal is to extract images that carry meaning relevant to the articles or content on the page. Most images used in articles, products, news, etc., are in PNG or JPEG formats. That's why these formats are prioritized. SVGs or animations are typically used for icons or buttons, which we don't prioritize for extraction.
To accommodate various preferences, we should make these settings customizable (treat them like hyper parameters). For now, use config.py in the project's root to keep all constants, like the list of prioritized image formats or thresholds. Import from there. Eventually, we can convert these settings to a YAML file for better flexibility.
Your second question was about using width and height. Some servers and applications can return different versions of the same image based on specified width and height, meaning two images with the same source value but different dimensions might actually be different. We can decide to keep the smallest one or both, but this was the reason for including width and height. If there is no such differentiation, then the source alone is sufficient.
Lastly, I recently posted a tweet reviewing Florence-2, a new vision model, which I've quantized to 4 bits. It's quite impressive. I recommend using this model instead of OpenAI or GPT. You can refer to my tweet and the accompanying Colab notebook, which explains how to use and experiment with it. If it performs well, I can further optimize it for our projects in the second phase. This is link to the post https://x.com/unclecode/status/1805992326915108996?s=46&t=J1hebTqzIYxu8ZpV-7GoyQ
Well done so far, and I look forward to your next updates.

@aravindkarnam Hope you are doing well, any update on your side?

@unclecode Ran into some health troubles over the weekend. Just recovered fully and got back to work today. I created a sample set of 25 URLs(sample.csv) with a good mix of sites. Ran into a lot of edge cases and exceptions with the ecommerce websites for the filtering and scoring functions. Fixed them all.
Now I ran the code I written for filtering, scoring and text extraction based on nearest eligible parent and have the result ready in pandas dataframe(Refer to data.csv. "desc" column is extracted text and "score" is calculated based on the approach we discussed above). Now I'll try to get Florence-2 model to evaluate the text I extracted from the nearest parent approach. I'll share my findings here.
I'm already committing in my forked branch. https://github.com/unclecode/crawl4ai/compare/main...aravindkarnam:crawl4ai:main. Please take a look at the code and let me know if any changes are to be made.

@aravindkarnam I'm so sorry to hear that. I hope you're already recovering and doing well. I was a bit busy today and couldn't go through your changes, but they sound very intriguing. I'll check them tomorrow morning and update you. I'm glad you created a good sample set and are now ready to go for the Florence 2 model. I'll review everything tomorrow morning and update you. Thank you so much. Take care of yourself, please.

@aravindkarnam I have just returned from a business trip and checked the data.csv file. The results are quite good; well done! It seems like the idea almost worked. I tested a few instances, and everything looks good. Let's move forward with Florence-2. I am very confident that we have found a good solution, and most importantly, a fast one. I have one suggestion regarding the code. The section where you filter out images without display and wrapped by a button, the three nested if statements bother me a bit. I have refactored your code, applied a bit separation of concerns and propose the following solution.
def is_valid_image(img, parent, parent_classes):
    style = img.get('style', '')
    src = img.get('src', '')
    classes_to_check = ['button', 'icon', 'logo']
    tags_to_check = ['button', 'input']

    return all([
        'display:none' not in style,
        src,
        not any(s in var for var in [src, img.get('alt', ''), *parent_classes] for s in classes_to_check),
        parent.name not in tags_to_check
    ])

def process_image(img, url, index, total_images):
    if not is_valid_image(img, img.parent, img.parent.get('class', [])):
        return None

    score = score_image_for_usefulness(img, url, index, total_images)
    if score <= 2:
        return None

    return {
        'src': img.get('src', ''),
        'alt': img.get('alt', ''),
        'desc': find_closest_parent_with_useful_text(img),
        'score': score,
        'type': 'image'
    }

# Main processing
imgs = body.find_all('img')
media['images'] = [
    result for result in
    (process_image(img, url, i, len(imgs)) for i, img in enumerate(imgs))
    if result is not None
]

The rest all looks fantastic, plz continue to finish up the Florence-2 then seems we are good to go. Good job!

Thanks @unclecode. I've trying to run florence-2 on my mac book m2 and haven't been able to. It keeps asking me for some nvidia dependancies(cuda/nvcc) that I'm not able to find a mac version of anywhere. For now, I'm resigned to try this florence model on collab notebooks itself. So I'll have to export the results from my local environment into a csv and then run the tests on collab. Please let me know, if you figured out how to run this florence notebook on a mac.

The section where you filter out images without display and wrapped by a button, the three nested if statements bother me a bit.
I'll update this bit. Thanks for the suggestion.

@aravindkarnam A few weeks ago, I created a Colab to review the quantized version. I quantized and uploaded the model to my HuggingFace repository. Please feel free to use it and determine if it provides better results. Ultimately, you can conduct your evaluation on Colab.

@unclecode I have tried your florence quantized version(8-bit model)unclecode/folrence-2-large-4bit. I've tried to load it directly with following code.
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("unclecode/folrence-2-large-4bit", trust_remote_code=True)

This has a dependancy on library flash_attn which further needs nvcc & CUDA in the environment, which currently is not possible with apple silicon. Both my macs have apple M2 chips.
I have been limited to use google colab since it had dependancy on GPU and CUDA either way (quantised or non-quantised model). I could only get sometime per day with T4 GPU backend, after that the kernels allocated doesn't have GPU and code errors out saying GPU is required. Therefore I couldn't proceed on this task as quickly as I wanted to. (I'm also not comfortable adding my card at this point either 😟, to get dedicated GPUs considering I'm new to this field and don't want to risk raking up huge bills unknowingly with a bad config).
Please let me know if there are any workarounds for this. (Maybe if you figured how to make these libraries work for apple M2 or there's some other free resources I could use)
Overall, I completed preliminary testing with florence. Here are my observations.
The results felt way way better with text scraped from immediate surroundings of image than model captioning.
For example the when scraped text says

Shaun Pollock. of. South Africa. bowls to. Michael Hussey. of. Australia. during the 2005. Boxing Day Test. match at the. Melbourne Cricket Ground."

the model captions it as

A group of people standing on top of lush green field.

In fact there's nothing wrong with model's caption that's exactly what's in the picture, but there's a cricket match going on there, which you will know only if you read the surrounding text and understood cricket.
Similarly another image from product landing pages the scraped text is

Cut Your Reading Time in Half. Let Speechify Read to You.. Gwyneth Paltrow. English Female Voice. Snoop Dogg. English Male Voice. John. English Male Voice. Mr. Beast.

whereas the model's caption is

phone with a message on the screen next to a phone with an image of a person on it

Another example from wikipedia scraped text is

Endeavour. docked at ISS during the STS-134 mission

and model caption is

image of a satellite in the sky with a building in the background"

An example from ecommerce, scraped text is

Dreo Humidifiers for Bedroom, Top Fill 4L Supersized Cool Mist Humidifier with Oil

and the model caption is

a glass jar filled with water sitting on top of a table

In almost all cases the model's caption would be right/true if you look at the picture, but there's way more going on in the picture and page context is required to truly understand what's in the picture. I've played around with prompt as well, but it doesn't seem to be possible for the model to clearly describe what's going on in the picture based on the picture itself.
Please let me know how do you advise to further proceed on this? We can either stop at this point or further refine the scraped text by passing it to model along with the image.
PS. I also made changes in this (commit)[3fb39cdb2e13bf81573bad21333c99247ad8e369] as per your (suggestion)[https://github.com/unclecode/crawl4ai/issues/26#issuecomment-2213505123]. If you can create a new branch for this feature, I can raise a PR for code I have so far.

@aravindkarnam, I've been away for a few days. I checked the details and agree we gather much better info than those models, and we do it much faster. Such a great result and experience, isn't it? We've had a complete professional/academic journey together. I created the branch "main-img-captionify," sent a pull request over there, and tested it. Excited to see it!

@unclecode Raised a PR ☝🏽. Learned a lot of stuff, while working on this enhancement. Looking forward to more such collabs.

@aravindkarnam  This is a great job; I'm currently bypassing image size checks, which slow things down, yet the scoring remains effective. I will merge this into the main branch and release new version 0.2.8, giving you full credit for your contribution. I'm also working on integrating PDF functionality without third-party libraries, and the results are promising. Another upcoming task is using multiprocessing to crawl multiple URLs, which will support a depth-first search approach for deep crawling. Initially, we'll fetch the sitemap and create one if it's unavailable, then crawl the entire page. I think this is a good area for collaboration if you're interested.

Another upcoming task is using multiprocessing to crawl multiple URLs, which will support a depth-first search approach for deep crawling. Initially, we'll fetch the sitemap and create one if it's unavailable, then crawl the entire page. I think this is a good area for collaboration if you're interested.

I'm interested unclecode. I was a developer who turned into product management career for past few years, who's now picking up on coding again as I'm now trying to build my own product as a solo founder. I believe this is the best way to build back my coding chops. So count me in for any exciting new enhancements your'e planning.
Also I suggest that we start a discord server for crawl4ai, atleast for collaborators and selected users so that we can communicate more efficiently on bugs and enhancements.

@aravindkarnam  It's nice to know your back story and understand the feeling. Being in product management must have given you some benefits in understanding different perspectives. Often, solo developers might not sense the broader picture, and project managers who aren't developers miss the developer's side. I'm curious to see how it goes for you with both experiences.
Let's get into Discord. I'll create a server and add you as a moderator. Also, I'm handling the pull request you sent.
There are certain things I want to bring to Crawl4AI. We can discuss deep crawling, creating snapshots of all website pages, and generating quick summaries. This will help create a semantic index for websites, useful for in-context learning for any large language model.
I'm also working on an interesting PDF library that I want to open source and bring to Crawl4AI. There's nothing more rewarding than building products that you also use. Good entrepreneurs often become users of their own products.
I'm an entrepreneur running a tech business in Southeast Asia for the last 8 years, with different divisions, offices, and a big team. Now, I have a bit more time, almost like early retirement. What I want to do is focus more on my computer scientist and researcher side, after these years, and it's so fun for me now working in the AI area. One side is relevant to my businesses, but the other side is about contributing more to open source, democratizing it, and working with people from other generations like yourself. So, you are most welcome.
I already have multiple projects that I typically tweet about and share on my Twitter/X account. I'll bring them to this Discord as well, and if you want to contribute, that would be great. These projects are somehow connected with each other too. So that's it. Let's build together.


Implemented Training Class + Add methods for convenience to sampling routines

CHANGELOG:

Addded __repr__() to many if the classes that we defined froweasier visualization during code-execution.
Added a base class mcmc_sampler_base which is a decorator class using which we can access the mcmc subroutines (both quantum and classical) . Added classes inheriting from the above to quantum-mcmc module and classical-mcmc module.
Refactorted training module with the previous changes.
Added a testing script for easier checking of bugs in implementation.

Made receent push with minor changes involving moving around the classes and capitalization .
Still to resolve the issue about  ..

In that spirit, maybe they should contain methods for sampling and getting the distribution. The run can be renamed to train where the training options are passed to, instead of the init of the class. As it is now, the class is a very fine wrapper around the training function. Could we rename them as BoltzmannMachine and the quantum assisted version? Then it is the top level class that handles all the sampling and training stuff. I think that makes most sense.


Add docker build pull option to devcontainer properties

It would be nice to be able to configure devcontainer.json properties to pass the docker build --pull option to "always attempt to pull a newer version of the image" for scenarios where the Dockerfile does not specify an image version or specifies latest.

There is F1 > Remote-Containers: Rebuild Without Cache and Open in Container command that will add --pull. Does that work for your case?


feat: add menu on macos

Summary of PR
Attempt to fix global copy paste on macOS.
Tests for unexpected behavior

none_whatsoever

Time spent on PR
5mins fix and couple hours of research

Did you manage to test this?  I ask only because the title of the PR indicates it may not work

Yes I built it locally and it worked. Copying I’m search wouldn’t work since we have custom copy keybinds but in settings it works as we expect it to.

Maybe package bump in this PR to save a commit so we can release this out


Stars tutorial frontend not rendering nodes

The stars application used in the network policy tutorial is not working. Specifically, the frontend is not rendering any nodes.
Expected Behavior
I expected to see three nodes representing the different components.

Current Behavior
The frontend is loading but no nodes show up.

No errors show up in the browser console.

This was tried in Chrome 93 and Firefox 94.
Possible Solution

Steps to Reproduce (for bugs)

Follow the AWS Guide.
Deploy stars application on cluster.
Port forward frontend service.
Load frontend in browser.

Context
I'm following the EKS Calico installation guide to install Calico on my EKS cluster for network policies using the AWS CNI.
After following the EKS guide and installing Calico, I launched the stars application to check that network policies were being enforced.
Your Environment

Calico version 3.13.4 (This version was installed for cluster compatibility reasons)
Orchestrator version (e.g. kubernetes, mesos, rkt): EKS 1.17
Operating System and version: Ubuntu 18.04
Link to your project (optional):

@AlejoAsd Any reason why you can't upgrade? Could you try the same tutorial on latest Calico version v3.21?
Please raise another issue if it's still not working. I doubt if we would fix this and deliver a patch release for v3.13.

Mostly due to current version freezes in the project. I will see if we can update Calico to 3.17 as it is the latest that my cluster version supports and get back to you.


[API] Support pug match(not team)

not for team,for 10 personal pug match

Related #134 . e.g. "is_pug" in "match" table.

Related question about this - the way I organized ten man's was making two teams (Pug Team One and Pug Team Two) and turned off get5_check_auths 0. Is there another way around this so stats are still tracked per player?

I think you can put steamids in form field on front-end, and POST it to API. Not sure if I chose this wat tho.

There's several way to implement pug,but every single choice has problem :

Create temporary team for each pug matches.
PRO:Back compatibility with get5-web
CON:API will generate teams in each match(but it also generates player_stats and round_stats for every matches, so it should'nt be big issue)
Plus I need to think about member picks(captain picks,random,ELO-based,manual.)

Add "players" column on match table
PRO:API won't generate team.
CON:We'll lost compatibility with get5-web cuz get5-web can't find teams.
get5-web will be bugged if it can't solve team.

Add "match_pug" table and manage pugs with that.
PRO:its completely separated with original "match" table, so it won't affect original matches. back-compatibility safe.
CON:implementing it is not easy(?)
Like match list(/matches) shows match by "ID DESC" order,it can't be mixed.

Thoughts? @PhlexPlexico

Hmmm, I think solution 3 would be the best to go if you want to keep backwards compatibility.
What I did (option 1) when organizing pugs through discord is I had two teams (Pug Team 1 and Pug Team 2) with empty player ID fields. When the selection through discord was done, it would assign each player into the given team, then do match setup. Once match is finished, it wiped away the teams from the player column (or should have). It was a very experimental feature I put in here and here.
That was my hack solution of fixing for pugs. I understand that get5 is not intended for pugs, but yeah. That was my workaround to play with the system as it currently was.
The good option with 3 is that if you have that match_pug table, you can always just call a union on the table, and even check if the table exists first, and display if it doesn't, that should fix some compatibility issues.
TL;DR - Option 1 is fastest and it has caveats, esp. with captain picks on front-end, can have two static teams. Option 2 is de-normalizing DB as well, would be against. Option 3 is most complete and can put in enough checks to keep backwards compatibility, but indeed hardest.
Hope that helps :)

Thanks! I'll think to go with 3rd way. There's lot of things to think about to do 3rd way :P

One other way of approaching this actually - send in an incomplete match structure to the game side, or just set the API key with the hostname, and get people to setup pugs in game (i.e. call /get5 from chat). Provided the match is created in the DB with the API key, it should have no issue in updating playerstats during the match, right? Then teams/etc can be determined during setup in the game... Always a possibility.

I think that also works,but ppl who knows server password can freely enter the game(get5_check_auths?).
and other players can't kick other guy if there's no admin.
but using icomplete structure approach is good imo,I'll think about it 👍

So coming back a year later here, I think I have figured out a good approach to this that takes care of tables from being clogged up. It's a bit messy on the DB side though, since it does allow some constraints to be changed.

Allow NULL values on match table team_ids. On delete team set null.
Bring over team names into team1/2_string field in the match table.
Create additional field labeled is_pug in match table (default 0/false). This can be used to track stats of players in PUGs only.
Before match create, create two teams (I do this via discord bots to select teams) and label it as "team_{captainName}".
Create match with is_pug flag set to 1
Send match structure to server (complete structure!).
On the series finish call on your API, check if is_pug is set to 1, delete teams from team table.

player_stats, team, map_stats, and match I believe will have to have loose restrictions on the team_id FK constraint to allow NULLs and on DELETE SET NULL.

Match is recorded, stats stay recorded, and you can now sort all your data based on PUGs as well.

The front end could have a "create pug" where a user can enter 10 Steam IDs and it randomizes the teams and captains as well.
Pros of this:

Still close to original get5 tables, one column added.
I think it should be backwards compatible with the original get5
Sortable data based on games being pugs or not
get5_check_auths can be enforced and no one should be able to enter the game.

Cons:

Looser database restrictions
Need to implement maybe another page that's "create pug".

This is the approach I use and I don't think I've found any issues with it so far. The workflow I have setup is:

Create teams
Create match
Do vetoes
Select Server
Send complete match structure
Delete from teams once match is complete

And it seems to work pretty well, the only thing increasing is the team ID

Sorry for late res, and thanks for nice proposal!
I was so inactive to maintain this project tbh, I was using a lot of my time to do for another project
actually thats great approach, back-compatibility and close to original structure(only one column).
I regret that I added so much functions even though project itself isnt so stable...
I'll think to implement this approach, and way to improve stability of this project xd
again thank you for proposal and focusing on get5-web-go, I'm promise to maintain this project once again!

duplicate #166


Cloneable Bundles

What problem does this solve or what need does it fill?
There are cases where I need to clone a bundle, but only if the components in the bundle are also cloneable.
Normally when you insert a bundle, the world takes ownership of the bundle and there is no need for cloning. However, a lot of folks are looking into reactive or dynamic frameworks where the "assembly" or "update" step happens multiple times. This update step might involve updating components, or attaching a copy of the components to a different entity. This means that the template needs to retain ownership of the bundle between updates.
What solution would you like?
What would be the most convenient for me is if Bundle could conditionally implement Clone. This would only happen if all of the components in the bundle are cloneable.
What alternative(s) have you considered?
It is currently possible to do this by inserting components one at a time instead of using bundles, since individual components can implement Clone.
We can also work around this problem by using a closure which generates a copy of the bundle each time; however the resulting code is quite ugly, especially in the case like bevy_mod_picking where the components being inserted also contain closures, so you get closures inside of closures.

Going to close this, as my research has taken a different direction and I no longer need this.


Update dependencies

Coverage remained the same at 85.562% when pulling 838f4b019ab73e14cc7cbba980b00c202022bf69 on rel_branch into e250784c76503ef66a75778a2b7e22270a68d1e0 on master.


Back button not working with inPageLinks with workaround

When enabling inPageLinks during initialization, the anchor links work but the back button takes you to the previous page (the page in your history before the page that contains the iframe)
Expected behavior:
Take you to the previous spot on the parent page where the link was clicked.
Workaround:
I added the following to the child page to add the hash to the parent page history. I also added the class "samepagelink" to make assigning the javascript easier. This adds the same hash from the child page link to the parent page history.
    document.addEventListener("DOMContentLoaded", function() {
        var elements = document.getElementsByClassName("samepagelink");

        Array.from(elements).forEach(function (anchor) {
            anchor.addEventListener("click", function (event) {
                window.parent.location.hash = anchor.hash;
            });
        });
    });

Interesting idea. Will give it some though on how to integrate it to the main code base.

This was a one line fix that should have been there from the start. Thanks for the idea. It will be included in the next release.


Handle tweet data with empty values

Example of image with empty values

From:

#111

Now it is showing fine??


APKs will be overritten with same name after build

If you generate APKs with the same name, the step will override them when it moves them to BITRISE_DEPLOY_DIR
🚀  Copy ./app/build/outputs/apk/app-release.apk to /bitrise/deploy/app-release.apk
🚀  Copy ./app/app-release.apk to /bitrise/deploy/app-release.apk

Exporting output: $BITRISE_APK_PATH => /bitrise/deploy/app-release.apk

Hi @birmacher ,
thanks for the report, the issue is solved in 1.5.0


add support for querying cpuid

Draft.
While I haven't tested this on a proper x86 machine (but I do have a dusty core duo laptop which may really be the thing I need, if I manage to turn it on :D) I have sketched a solution by putting together everyone's feedback (including @ncruces' 🚀). In the meantime I have also discovered that GOARCH=amd64 has the ABM feature flag off on Rosetta2.

this is in a new cupid_amd64.{s,go} because I felt it would pollute too much impl and arch didn't feel like the right place (that's for runtime)
For now I have only added the constants that we use and/or plan to use (e.g. SSE3, 4.1, 4.2), and only the related bitmaps
you still have to carefully pick the right set of flags because there is no proper type checking in place (all are uint64). Suggestions welcome for better ergonomics/idioms

Values are kept in a struct that is local to the compiler instance. This is re-inited at for each amd64Compiler -- TBF it may be created once and then copied over when necessary. I wanted to keep it local to the compiler instance, instead of global, to make sure we can properly test it.
Speaking of testing: the compiler test right now is kind of silly because it checks for the length of the generated byte array. The one reason is that I couldn't figure out how to decode or inspect the list of opcodes the right way. This is the main reason why this is draft.

I tried this on the affected machine and I can report that it worked:

[x] wazero tests pass
[x] wazero-sqlite3-bug tests too

I am not 100% sure I am doing the right thing here 😬
in your PR you are and-ing 0x0400_0000, in this PR it's 0x20. For some reason on my (dusty old and excruciatingly slow) machine this still prints "100000000..." instead of "2.500..."; and it only works with 0x0400_0000.
However, according to the spec the 5th bit on ECX should be 0x20 🤪

I am not 100% sure I am doing the right thing here 😬
in your PR you are and-ing 0x0400_0000, in this PR it's 0x20. For some reason on my (dusty old and excruciatingly slow) machine this still prints "100000000..." instead of "2.500..."; and it only works with 0x0400_0000.
However, according to the spec the 5th bit on ECX should be 0x20 🤪

OK, then I need to recheck. Did the tests pass for me because of caching, or because I have both 0x20 and 0x0400_0000?
I did notice that. From the docs, I assumed it's the 5th bit from the left: ((1<<31) >> 5) == 0x0400_0000, which did the right thing on my machine. You assumed the opposite (1 << 5) == 0x20.
I thing we need to figure it out from other implementations.

I think, based on the Intel "sponsored" library, the correct implementation is to first get leaf0x80000000 (which I didn't do, to make sure you can get leaf0x80000001) then get leaf0x80000001.
https://github.com/aregm/cpuid/blob/219e067757cbc0ddb71908b19826ddca4177987d/cpuid_amd64.go#L218-L229
Then compare extraFeatureFlags with 0x20.
This is HasExtraFeature: https://github.com/aregm/cpuid/blob/master/cpuid.go#L101-L104
This is ABM: https://github.com/aregm/cpuid/blob/219e067757cbc0ddb71908b19826ddca4177987d/cpuid.go#L462-L467
So it seems you're right, and I messed up, but maybe your excruciatingly slow computer needs to check leaf0x80000000 before doing leaf0x80000001?

OSS collaboration like this makes my day, thanks @ncruces @evacchi @mathetake!


jmove commons to unit-5

the only change is to move commons project to junit-5

Thanks let me know when it is ready

@ryanjbaxter ready


[cloud_firestore] CollectionRef.snapshots.listen leaks

I have a screen that subscribes to a collection using code like:
  @override
  void initState() {
    super.initState();
    Firestore.instance
        .collection('MainThing')
        .document(documentId)
        .collection('subcollection')
        .snapshots()
        .listen(_processUpdates);
  }

  void _processUpdates(QuerySnapshot snapshot) {
    // process the snapshot...
    setState(() {
      // update state
    });
  }

I noticed that when I close this screen, and then go back to it, I get an exception like:
[VERBOSE-2:ui_dart_state.cc(148)] Unhandled Exception: setState() called after dispose(): _DashboardScreenState#a81a3(lifecycle state: defunct, not mounted)
This error happens if you call setState() on a State object for a widget that no longer appears in the widget tree (e.g., whose parent widget no longer includes the widget in its build). This error can occur when code calls setState() from a timer or an animation callback.
The preferred solution is to cancel the timer or stop listening to the animation in the dispose() callback. Another solution is to check the "mounted" property of this object before calling setState() to ensure the object is still in the tree.
This error might indicate a memory leak if setState() is being called because another object is retaining a reference to this State object after it has been removed from the tree. To avoid memory leaks, consider breaking the reference to this object during dispose().
#0      State.setState.<anonymous closure> (package:flutter/src/widgets/<…>

And this is on the setState inside _processUpdates.  I believe what is going on is that I now have two snapshot listeners.  My original screen that triggered the first snapshot listener still has an active Stream linked to the _processUpdates method of the now unmounted screen.
This suggests that I've got a leak! 🚨 And to address it, I believe I need a way to cancel the listener.  Alas, the snapshots method returns a Stream, not the StreamController, so I do not have access to a close method.
I am using Flutter 1.9.1+hotfix.6, Dart 2.5.0, and Cloud Firestore 0.12.9+6.

Whoops!  My bad!
I need to retain the StreamSubscription from the listen and then cancel that upon dispose.

Hello @awhitford
Can you please tell me how to StreamSubscription from the listen? I also facing same issue, can you please help me about this?

Hi @SwapnilDaware7 , I'm just doing something like this in a StatefulWidget:
  StreamSubscription<QuerySnapshot> _subscription;

  @override
  void initState() {
    super.initState();

    assert(null == _subscription);
    _subscription = Firestore.instance
        .collection('MainThing')
        .document(documentId)
        .collection('subcollection')
        .snapshots()
        .listen(_processUpdates);
  }

  @override
  void dispose() {
    assert(null != _subscription);
    _subscription.cancel();
    _subscription = null;

    super.dispose();
  }

Essentially, the listen method is returning a StreamSubscription which needs to eventually be canceled, which I do in dispose.
Does that help?

This is very strange. In my case I'm not calling setState((){}) in anywhere but the same error appears. I tried adding a check for mounted property of the widget but that doesn't solve the problem either.


fix for issue 23

this fixes the no icons issue mentioned here https://github.com/Purfview/Discogs-Scout/issues/23

But there is no changes in your PR.

Now check. Should be updated


All I2C Messages Drop on XIAO BLE Sense

Describe the bug
NRFX_TWIM Fails on the XIAO BLE Sense for seemingly all I2C read/writes. Currently I am only trying to read a single byte from a register over I2C on the XIAO BLE Sense. The register is on the LS6DS3TR-C, but the I2C issues seem to be unrelated to the register I am trying to read.
To Reproduce
Build the fs shell sample with west for the xiao_ble_sense board. To have the i2c devices show up you first need to sent CONFIG_I2C=y.
The command I am running to build is: west build -p -b xiao_ble_sense samples/subsys/shell/fs/
Try running any I2C commands, as all messages seem to be dropping. Specifically I am trying to read the WHO_AM_I register (0x0F) on the LSM6DS3TR-C (address 0x6a), which should always read 0x6a. I am doing this with the following command:  i2c read_byte i2c@40003000 0x6a 0x0f
Note: i2c@40003000 is the internal i2c bus that the LSM6DS3TR-C is on. The LSM6DS3TR-C is powered by a regulator, but the default XIAO BLE Sense device tree has it configured to turn on at boot. I did run an i2c scan on the external i2c bus and the messages also all seemed to be dropped according to log messages.
If reproduced you should see: "Failed to read from device: 0x6a"
Expected behavior
It should simply read 0x6a from the register.
Logs and console output
Here are the logs from a fresh boot. Only changes from a fresh west init were enabling CONFIG_I2C, enabling I2C debug level logging, and enabling NRFX debug level logging. I included the device list output in addition.
[00:00:00.295,684] <inf> NRFX_TWIM: Function: nrfx_twim_init, error code: NRFX_SUCCESS.
[00:00:00.295,715] <inf> NRFX_TWIM: Instance enabled: 0.
*** Booting Zephyr OS build zephyr-v3.3.0-1420-g9889e8e20a4f ***
uart:~$ device list
devices:
- clock@40000000 (READY)
- gpio@50000300 (READY)
- gpio@50000000 (READY)
- cdc-acm-uart (READY)
- uart@40002000 (READY)
- flash-controller@4001e000 (READY)
- i2c@40004000 (READY)
- i2c@40003000 (READY)
- spi@40023000 (READY)
- spi@4002f000 (READY)
  requires: gpio@50000000
- lsm6ds3tr-c-en (READY)
  requires: gpio@50000300
- p25q16h@0 (READY)
  requires: gpio@50000000
  requires: spi@4002f000
uart:~$ i2c read_byte i2c@40003000 6a 0f
Failed to read from device: 6a
[00:00:17.733,612] <inf> NRFX_TWIM: Transfer type: XFER_TX.
[00:00:17.733,642] <inf> NRFX_TWIM: Transfer buffers length: primary: 1, secondary: 0.
[00:00:17.733,673] <dbg> NRFX_TWIM: nrfx_twim_xfer: Primary buffer data:
[00:00:17.733,673] <dbg> NRFX_TWIM: nrfx_twim_xfer:
                                    0f                                               |.
[00:00:17.733,703] <dbg> NRFX_TWIM: nrfx_twim_xfer: Secondary buffer data:
[00:00:17.733,734] <dbg> NRFX_TWIM: nrfx_twim_xfer:
[00:00:17.733,795] <wrn> NRFX_TWIM: Function: nrfx_twim_xfer, error code: NRFX_SUCCESS.
[00:00:18.233,856] <err> i2c_nrfx_twim: Error on I2C line occurred for message 0
[00:00:18.233,886] <inf> NRFX_TWIM: Instance disabled: 0.
[00:00:18.234,008] <inf> NRFX_TWIM: Instance enabled: 0.

I believe the warning message from nrfx_twim_xfer where it says error code: NRFX_SUCCESS is a red herring. Unsure why that logging message is even in there but it logs the warning unconditionally at the end of the function, even if the return code is NRFX_SUCCESS.
Environment (please complete the following information):

OS: Arch Linux
Toolchain: Zephyr SDK (I think, just using west)

Additional context
I initially created this issue here: https://github.com/zephyrproject-rtos/zephyr/issues/55892 believing it to be an issue with the sensor. But the more I looked into it the more it seemed to be the nrfx_twim driver.
Some notable lines:
This is where the Zephyr driver always fails. It fails to take the completion_sync semaphore. https://github.com/zephyrproject-rtos/zephyr/blob/b0ea0c8ec6135aa5cb8d0e70ca72af2abe1f38fa/drivers/i2c/i2c_nrfx_twim.c#L155-L178
This should be released by in the event_handler here: https://github.com/zephyrproject-rtos/zephyr/blob/9889e8e20a4ffe1f442f499d51e55fa7cb73c806/drivers/i2c/i2c_nrfx_twim.c#L215-L235 But this event handler doesn't seem to be being called, as I have added logging into it and didn't get any output.

I'm going to guess that this board has no pull-up resistors and is assuming that the SoC enables them, but isn't set up to do that. Try adding this to the pinctrl files for the board for the i2c instance you are using for the default mode:
    bias-pull-up;

I tried adding bias-pull-up; to the i2c0_default node (left sleep pin alone) as follows but saw no change in the log output or behavior:
	i2c0_default: i2c0_default {
		group1 {
			psels = <NRF_PSEL(TWIM_SDA, 0, 7)>,
				<NRF_PSEL(TWIM_SCL, 0, 27)>;
			bias-pull-up;
		};
	};

which according to the xiao_ble_sense.dts is the assigned pin for default
https://github.com/zephyrproject-rtos/zephyr/blob/d44e36d4388dc20a0f66ba6b742efb5356a25238/boards/arm/xiao_ble/xiao_ble_sense.dts#L24-L39

Issue was the same as the issue specified here. I came across this article before but dismissed it since the poster mention his I2C messages would work sometimes, but not all the times. Anyways the fix is simply:
--- a/boards/arm/xiao_ble/xiao_ble_sense.dts
+++ b/boards/arm/xiao_ble/xiao_ble_sense.dts
@@ -7,6 +7,7 @@
 /dts-v1/;
 #include "xiao_ble_common.dtsi"
 #include <zephyr/dt-bindings/i2c/i2c.h>
+#include <zephyr/dt-bindings/gpio/nordic-nrf-gpio.h>

 / {
        model = "Seeed XIAO BLE Sense";
@@ -14,7 +15,7 @@

        lsm6ds3tr-c-en {
                compatible = "regulator-fixed-sync", "regulator-fixed";
-               enable-gpios = <&gpio1 8 GPIO_ACTIVE_HIGH>;
+               enable-gpios = <&gpio1 8 (NRF_GPIO_DRIVE_S0H1 | GPIO_ACTIVE_HIGH)>;
                regulator-name = "LSM6DS3TR_C_EN";
                regulator-boot-on;
                startup-delay-us = <900>;

Since you found a solution we will close this issue. Feel free to send a Pull Request with your suggested fix.

PR created here: https://github.com/zephyrproject-rtos/zephyr/pull/56640


[base-ui][NumberInput] Fix input value resetting on Focus

closes: https://github.com/mui/material-ui/issues/40340
before: https://codesandbox.io/p/sandbox/sad-elion-zmhtvz?file=%2Fsrc%2FApp.js%3A11%2C6
after:  https://codesandbox.io/p/sandbox/gallant-haze-rvcnm4
Below piece of code is responsible to update dirtyValue state when vaule changes through onChange handler, but updating dirtyValue is missed if value is updated through useEffect like shown in issue. So updating dirtyValue whenever value changes seems to fix the issue.
https://github.com/mui/material-ui/blob/0fcc3e053e8cd8f13769485a0f6d587e58fa90cf/packages/mui-base/src/unstable_useNumberInput/useNumberInput.ts#L170-L180

[x] I have followed (at least) the PR section of the contributing guide.

@sai6855 Thanks for investigating this 🙏
I'm about to replace the setStates though in https://github.com/mui/material-ui/pull/40206 😓
For this issue, I've adapted your fix: https://github.com/mui/material-ui/pull/40206/files#r1438284448

@sai6855 Thanks for investigating this 🙏
I'm about to replace the setStates though in #40206 😓
For this issue, I've adapted your fix: https://github.com/mui/material-ui/pull/40206/files#r1438284448

Ah i see, no issues for porting the fix 👍 .
Closing this PR as you ported the fix


Can't get the actual path of image where it is stored in my internal storage[🌟]

Description
Explain the feature request in as much detail as possible.
Proposed Solution
What needs to be done to make it happen?
Test Solution
How would we test & validate the feature?

I also found this issue during my Android App development.
Currently, the library seems to create a new (resized) image from the selected image.
It might be nice if there is a way to access the image through fileURI, not contentURI.
https://github.com/react-native-image-picker/react-native-image-picker/blob/4bba2470452cd91a16e53049ab8c965b111acbb6/android/src/main/java/com/imagepicker/ImagePickerModule.java#L133-L134

I also found this issue during my Android App development.
Currently, the library seems to create a new (resized) image from the selected image.
It might be nice if there is a way to access the image through fileURI, not contentURI.
https://github.com/react-native-image-picker/react-native-image-picker/blob/4bba2470452cd91a16e53049ab8c965b111acbb6/android/src/main/java/com/imagepicker/ImagePickerModule.java#L133-L134

@sameerahmed56 what is the problem in using the returned uri on android and ios? Elaborate on the problem you'r facing.

@sameerahmed56 what is the problem in using the returned uri on android and ios? Elaborate on the problem you'r facing.

this came into crashlytics today:
Fatal Exception: java.lang.RuntimeException: Failure delivering result ResultInfo{who=null, request=13002, result=-1, data=Intent { dat=content://com.google.android.apps.photos.contentprovider/-1/1/content://media/external/images/media/963/ORIGINAL/NONE/image/jpeg/1882044403 flg=0x1 clip={text/uri-list U:content://com.google.android.apps.photos.contentprovider/-1/1/content%3A%2F%2Fmedia%2Fexternal%2Fimages%2Fmedia%2F963/ORIGINAL/NONE/image%2Fjpeg/1882044403} }} to activity {com.XXX/com.XXXX.MainActivity}: java.lang.NullPointerException: uri
       at android.app.ActivityThread.deliverResults(ActivityThread.java:4544)
       at android.app.ActivityThread.handleSendResult(ActivityThread.java:4586)
       at android.app.servertransaction.ActivityResultItem.execute(ActivityResultItem.java:49)
       at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:108)
       at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:68)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1864)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loop(Looper.java:205)
       at android.app.ActivityThread.main(ActivityThread.java:6991)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:884)

i'm using 3.2.1

Hi, I am having this problem as well. I feel like making another issue is pointless. My issue is that the tesseract library (https://github.com/jonathanpalma/react-native-tesseract-ocr) I am using to do OCR cannot handle the in app path, I get a file not found exception, when passing in the path to internal storage manually the issue is resolved and the library works.

Now latest version provides file uri to images. I hope the issue is solved, if not reply here or create new issue with details I am happy to fix it.


use log::error shows std::error

The only public log::error is a macro: https://docs.rs/log/0.4.14/log/macro.error.html. There is a private use std::error, but it isn't publicly visible: https://docs.rs/log/0.4.14/src/log/lib.rs.html#287

Cannot reproduce locally. Can you provide the source code needed to reproduce this?

I can reproduce it with current nightly in the rust-analyzer crate

Huh, yeah. Seems like it doesn't reproduce with a minimal crate that just pulls in log.

I think the issue here is that NameRefClassification resolves the path via Semantics which doesn't check visibility and just picks the first thing it finds in types -> value -> item -> macros order? https://github.com/Veykril/rust-analyzer/blob/052fe491797027ca930c0cd3d9cc62c920e2a17d\crates\hir\src\source_analyzer.rs#L514-L516
I'm not too familiar with path resolution

Repro as a goto_definition test:
    #[test]
    fn goto_def_avoids_private_names() {
        // Regression test for https://github.com/rust-analyzer/rust-analyzer/issues/7903
        check(
            r#"
//- /main.rs crate:main deps:log

use log::error$0;

//- /log.rs crate:log deps:core
use core::error;

#[macro_export]
macro_rules! error {
           //^^^^^
    () => {}
}

//- /core.rs crate:core
pub mod error {}
        "#,
        );
    }

I ran into this issue yesterday.


Update handling of web-link repositories in the UI

Update code base to keep weblink repositories in the submission's repo list and NOT in the metadata blob, in which it will fail metadata schema validation

Changes required:

[ ] components/workflow-metadata
[ ] components/workflow-review
[ ] service/metadata-blob
[ ] service/submission-handler
[ ] submissions/detail

[ ] controller
[ ] template


Is there any roadmap that support real device?

As the title

Hi @antstudio-git,
Beg your pardon, I somehow missed this ticket.
xcmonkey depends on idb, which is, unfortunately, limited to iOS Simulators in terms of accessibility automation. At least at the moment. See this documentation for more details. Hope this helps.
Best wishes,
Alexey


Stubs generated from Restdocs tests end up only matching double url-escaped query string values

Spring Cloud Hoxton.SR1
Contracts 2.2.1.RELEASE
With Rest docs, the wiremock stubs queryParameters values get stored url escaped even though Wiremock escapes them again to check against request parameters.
Sample
In a SpringCloudContractRestDocs test, a query param of .param("seasoning","salt,pepper") generates a stub like
          "equalTo" : "salt%2Cpepper"
           }
       }

The actual request to the stub server has to be double escaped to match, so ?seasoning=salt%252Cpepper

Could you please create a small sample that replicates this issue?

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.


[CT-2038] Detect breaking changes to column names and data types in state:modified check

If a model with contract:true has its column name/data types modified in a way that represents a breaking change, dbt should raise an error as part of the state:modified check.
What constitutes a breaking change?

Removal of column
name change of column
data_type change of column
changing contract:true to contract:false

Do we want this exception to be limited to commands where the user has explicitly selected state:modified? Or as soon as previous state is established? Although it feels a bit odd, I think I'm okay with raising it only when state:modified is selected.
We could also add a new sub-selector, state:modified.contracts.
Implementation detail: We have a number of same_* methods (example) which power state:modified selection today. It could make sense to define a same_contract method, with these return values:

True if the contract is the same (including models without contracts)
False is the contract has changed, in a non-breaking way
Raises an exception if the contract has changed in a breaking way

error message direct users to create a new version and point to versioning docs: https://github.com/dbt-labs/docs.getdbt.com/issues/2841

What about order? If the order of the columns changes should it be considered modified or not?

If contracts are set to false in both old and new states, then I assume that even if the the columns names or data_types change, that is not a breaking change?

What about order? If the order of the columns changes should it be considered modified or not?

For now, we're going to say that column order doesn't matter. We might add this as a configuration option later on if lots of users request it.

If contracts are set to false in both old and new states, then I assume that even if the the columns names or data_types change, that is not a breaking change?

Correct! Adopting the terminology / data structure proposed in #7184:

If old contract: {enforced: false} + new contract: {enforced: false} → never a breaking change
If old contract: {enforced: false} + new contract: {enforced: true} → never a breaking change
If old contract: {enforced: true} + new contract: {enforced: true} → detect breaking changes, following the logic in the description
If old contract: {enforced: true} + new contract: {enforced: false} → always a breaking change!

Currently when old has contract false and new has contract true, it will show as modified because the config has changed: unrendered_config = {'contract': True, 'materialized': 'table'}

Yes! It's definitely a change, and the model should be selected to run by state:modified / in CI. It's just not a breaking change, so it shouldn't raise an error.

@MichelleArk I've removed including "constraints" in the checksum, since it looks like that's not part of this ticket. Let me know if you want something different.


fix: bootstrapScript in values.yaml of the helm chart

SUMMARY
The bootstrap script had Syntax Errors and unnecessary nano installation.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
cd helm/charts
helm install .

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@craig-rueda Could you please review?


[FLINK-15081][docs-zh]Translate "Concepts & Common API" page of Table…

What is the purpose of the change
Translate "Concepts & Common API" page of Table.
Brief change log
Translation work has been finished without code annotation.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving):  no
The serializers:  no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature?  no
If yes, how is the feature documented? not applicable

Hi @libenchao , could you help to review this?

@CrazyTomatoOo could you resolve the conflict if you have time?

@libenchao thanks for your advise, I will revise it as soon as possible.

@wuchong Sure

Hi @libenchao, I have revise the doc, what should I do now?

@CrazyTomatoOo Thanks for the updating. It seems that your pr is conflicted with master, could you rebase your pr towards master to resolve the conflict?

@libenchao，I'm not sure if it's the right way to resolve the conflict. It's my first time. Please check it out. Thanks！

@libenchao, finished.

LGTM. +1 for merging. @CrazyTomatoOo let's wait for a committer to have a look.

I will review and merge it later.


Issue when there is space character inside RTA code #14

The SUMMARY_ROW_RE regex fails when there are RTA codes like the second and third row. It ends up reading folio number along with the first three characters of RTA code resulting in a null repsonse when ISIN check is done.

@codereverser The fix for this is to modify the SUMMARY_ROW_RE regex to the below regex:
SUMMARY_ROW_RE = (
    r"([\d/\s]+?)(?=\s\d)\s+([\s\w]+)-\s*\d*\s*(.+?)\s*([\d,.]+)\s*"
    r"(\d{2}-[A-Za-z]{3}-\d{4})\s*([\d,.]+)\s*([\d,.]+)\s*(\w+)\s*$"
)

I tried making an MR, but couldnt. Can you check this out and verify this fix?

many thanks for the report and fix. Shall verify this and release next version asap.

thanks for the fix. new version released.

Thanks!


use i18n default date format

Hello again @michelegera 👋 What do you think?
Instead of a hardcoded date format re-use I18n date format.
📖 http://guides.rubyonrails.org/i18n.html#adding-date-time-formats

Hi @michelegera 🙃
Friendly push, what do you think about this addition? 🤓

Thanks @neumayr! I’ll be releasing a new version of the gem today.


feat: allow configuring multiple package sources

Replaced the Discovery function with a pluggable IPackageSource API,
and providing an NpmJs as well as CodeArtifact source. These sources
can optionally be passed into the ConstructHub constructor to register
alternate package sources (by default, an NpmJs source is registered).
This unfortunately involved relativelu wide-spread changes in the
codebase, as some features had to be made visible from the public API of
the module, and some code needed to be re-used (hence, refactored to a
different location).
Fixes #281

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

For my own sanity, an IPackageSource is basically something that listens for relevant packages, then ingests their tarball and publishes an event to a queue describing where those assets are for consumers to do what they will. Those messages follow the package discovery schema. Tell me if I'm way off base but seems self-explanatory.

This is correct :)


Inline editing in record block

Inline record block editing for record page

[x] Add configurator option for inline record editing for record block
[x] Diplay the edit icon on hover for each field in the record block
[x] Display modal to edit field when edit icon is clicked
[x] Refresh record block when record field value is update

for today: study code, continue working on task and submit for code review

CL added


merge feature/add-translation-for-alert-wip into develop

:tada: This PR is included in version 2.5.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Move process specific settings to process

This moves process specific settings like caps, apparmor, and selinux
process label onto the process structure to allow the same settings to
be changed at exec time.
Signed-off-by: Michael Crosby<EMAIL_ADDRESS>

LGTM

LGTM


add k8s.gcr.io/cluster-autoscaler:v1.12.7 mirror

Hi @guohenghaha
please do not update required-images-mirrored.txt , update
https://github.com/nwcdlabs/kops-cn/blob/master/mirror/required-images.txt instead.

Hi
If you are trying to kubectl apply this
https://github.com/kubernetes/autoscaler/blob/8303a2355e55dc7a416a8a80359207b774556d51/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-one-asg.yaml#L132
k8s.gcr.io images are all mirrored in docker hub at https://hub.docker.com/u/googlecontainer
so you can change the image path to googlecontainer/cluster-autoscaler:v1.12.3 and kops-cn will use the docker hub china proxy to pull this image
https://github.com/nwcdlabs/kops-cn/blob/54f7de28cb98f1a912696850fee84f203873bf2d/spec.yml#L8
so you don't have to update the required-images.txt file

And please check this FAQ
https://github.com/nwcdlabs/kops-cn/issues/107


[FEATURE REQUEST][0.5.x] Back-merge "long lasting certs for Kubernetes" (#1302)

Is your feature request related to a problem? Please describe.
Clients use version 0.5.x with product version certified on this version, and would like to update and extend certs validity.
Describe the solution you'd like
Back merge and verify feature that was done in #1302 to 0.5.x branch.
Describe alternatives you've considered
Documentation can be a poor alternative.
Additional context
Product Team request.

These changes have been already backported, PR #1796 - https://github.com/epiphany-platform/epiphany/blob/v0.5/CHANGELOG-0.5.md


Color Sensor V3 Testing and Calibration

Control Panel
On the control panel, there are 4 different colours that appear twice each. To detect which colour is currently under the Control Panel Sensor, we are using the Rev Robotics Color Sensor V3.
Color Sensor V3
This sensor returns 4 channel RGBA( ( 0-255 ), ( 0-255 ), ( 0,255 ), ( 0-255 ) )  and a proximity value ( 0- 10).
The sensor also has an LED that can be toggled. It is a white LED that shines on the object/colour it is facing. Find out if this LED does any good or just makes things harder.
Get outputs from the sensor and use its data along with the colour comparison utility to detect which color is on the wheel.
This is being worked on in color-sensor branch

Can you add a shuffleboard widget with a slider to adjust epsilon for color comparison?

Make a new shuffleboard tab called "Configuration" for the widget

Here is the docs for sending a number to shuffleboard:
https://docs.wpilib.org/en/latest/docs/software/wpilib-tools/shuffleboard/layouts-with-code/sending-data.html#sending-simple-data
There is a bit that calls setDouble, I think you can use getDouble instead to read the value.
Also, instead of calling add for the widget, call addPersistent so the the value will stay across reboots.

Just thought about it. The threshold value should actually be stored on the robot, not the laptop.
Here is the API doc for doing this:
https://first.wpi.edu/FRC/roborio/release/docs/java/edu/wpi/first/wpilibj/Preferences.html

I am going to close this. Any updates on the sensor and its mech should be made into new issues


Always restart Goth on settings load

Fix an issue where Goth fail to load credentials on application start due to Goth app has already been started and env never got populated there.

Not sure if it will actually fix the issue, but this is the only one I can think about.


feat: bbs createKey, sign and verify

READY FOR REVIEW
Almost every change in this PR will be reverted when support for aries-askar is added, as we can then do all the functionality inside the wallet instead of the AFJ level.

Adds bls12381g1 and bls12381g2 to indyWallet.createKey
Adds bls12381g2 to indyWallet.sign
Adds bls12381g2 to indyWallet.verify
Store and retrieve the keyPair in the wallet via the public key
Added tests for the indy wallet

Codecov Report

Merging #684 (5af7328) into main (3713398) will decrease coverage by 0.04%.
The diff coverage is 84.07%.

@@            Coverage Diff             @@
##             main     #684      +/-   ##
==========================================
- Coverage   85.94%   85.90%   -0.05%
==========================================
  Files         342      343       +1
  Lines        7366     7451      +85
  Branches     1139     1163      +24
==========================================
+ Hits         6331     6401      +70
- Misses       1034     1049      +15
  Partials        1        1

Impacted Files
Coverage Δ

packages/core/src/modules/dids/domain/index.ts
100.00% <ø> (ø)

...odules/dids/domain/key-type/getSignatureKeyBase.ts
100.00% <ø> (ø)

...ges/core/src/modules/dids/domain/key-type/index.ts
100.00% <ø> (ø)

.../src/modules/dids/domain/key-type/keyDidMapping.ts
90.90% <ø> (ø)

packages/core/src/wallet/IndyWallet.ts
69.74% <72.22%> (+0.46%)
:arrow_up:

packages/core/src/crypto/BbsService.ts
91.17% <91.17%> (ø)

packages/core/src/crypto/JwsService.ts
91.66% <100.00%> (+0.75%)
:arrow_up:

packages/core/src/crypto/Key.ts
100.00% <100.00%> (ø)

packages/core/src/crypto/index.ts
100.00% <100.00%> (ø)

packages/core/src/crypto/multiCodecKey.ts
92.30% <100.00%> (ø)

... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3713398...5af7328. Read the comment docs.

Not quite sure what happened to the commits, but since we squash it is okay. (every commit is double)

I looked at the W3C PR and there are parts of this PR missing in that PR.
I think we want to merge this into a next branch for now. What do you think? Then the w3c pr can also target the next branch. Don't want to add all these dependencies in 0.2.0 if not used yet. Maybe we'll be able to solve this issue (#730) before 0.3.0 release so users are not required to bundle the react native bbs signatures if they're only using indy credentials for example.

I agree with not adding it in before the 0.2.0 release. We can merge it after that release or move it to the next branch. I do not really have a big preference.

@TimoGlastra base is now next.


Preliminary work on Scala 2.12 update.

Default on 2.12.0 build.
Updated dependencies.
Reverted machinist syntax to pre-cats-kernel notation.
Updated build plugins and SBT.

Remaining blocking points:

optimizer settings: the inliner should be enabled only for production builds. Has anybody a project.sbt example on how to do so?

-Yinline-warnings is not available on the latest scala-js. Is this option needed (thus it should be conditionally enabled), or can we remove it?

some macro patterns have changed in 2.12. For example, scala.this.Predef.intWrapper($i).until($j) becomes scala.Predef.intWrapper($i).until($j). How to deal with these changes? Special 2.12 code? Small compatibility layer?

Thanks @andrewsmartin and @xuwei-k for preliminary work on this in #600 and #605.

@denisrosset This is looking good so far! Seems like some changes in Scalacheck are causing compilation issues.
For the -Yinline-warnings, you could probably add it here: https://github.com/non/spire/blob/master/build.sbt#L233
Those settings apply to the JVM only, so shouldn't cause problems w/ scalajs.
What do you mean about the inline only production builds? Right now we use -optimize in JVM builds, which should also enable inlining. Do we want to do something different?
For the last issue, I guess we could just add it to our existing compat layer stuff we have w/ the scala_2XX source dirs. There should be one for 2.12 that exists already, from Seth's PR.

Regarding 1), I quote the Scala release notes:
We recommend enabling inlining only in production builds, as sbt’s incremental compilation does not track dependencies introduced by inlining.

@denisrosset Got it! Let's create an issue for 1), but get the rest of the PR merged. SG?

@tixxit: do you know the logic behind spire.macros.compat and HasCompat?
in https://github.com/non/spire/blob/master/macros/src/main/scala/spire/macros/Syntax.scala#L77

@denisrosset I don't know the reason for that - maybe @non can help?

@tixxit let's see. I added the scala.Predef compatibility hack to compat, without knowing exactly the import logic. It works, that's all.
I'm pushing some changes to adapt for Scalacheck. Three tests are failing, I have to find the mistake.

Current coverage is 48.57% (diff: 60.00%)

Merging #606 into master will increase coverage by 0.02%

@@             master       #606   diff @@
==========================================
  Files           181        181
  Lines         11471      11432    -39
  Methods        9811       9817     +6
  Messages          0          0
  Branches       1616       1615     -1
==========================================
- Hits           5569       5553    -16
+ Misses         5902       5879    -23
  Partials          0          0

Powered by Codecov. Last update 6beb315...a300079

Okay, after updating Travis to JDK8, the CI should pass.
Will try to remove the deprecation warnings in build.sbt, and we should be good to go.

@denisrosset Does Spire still build on Java 7?

@tixxit, did we already drop Java 6?
Can we do conditional CI so that 2.11 is tested on JDK 6 or 7, and 2.12 on JDK8 ?

Ah, that looks great! (latest commit)

👍  (assuming tests pass)

java.lang.NoClassDefFoundError: scoverage/Invoker$
	at spire.math.UShortTest.<init>(UnsignedTest.scala:221)

Crap.
Some also have the same error:
http://blog.ssanj.net/posts/2016-01-14-scoverage-breaks-with-java-lang-NoClassDefFoundError-scoverage-Invoker.html
and recommend:

@alexflav23 recommends turning off coverage when publishing or packaging:
You need to disable coverage with coverageOff before publishing or packaging and everything will be just fine. I’ve ran into this issue myself and that was the quick fix.

Ah, that looks great! (latest commit)

Shamelessly stolen from cats.

Ugh - these keep timing out failing w/ BS errors. I think one of the prime tests also tends to hang on ScalaJS :\


fix typo wording dinamsi to dinamis

I signed it

/assign @irvifa

@hadianr can you please sign the cla first?

/check-cla

/retest

I signed it

@irvifa done

Okay thanks!
/lgtm
/approve


Recommended resource minimums and autoscaling setup?

Is your request related to a problem? Please describe.
We are facing pretty bad latency when using this server and planning 20 different workspaces at the same time. Even for a single workspace it's pretty slow. Noticed that the ecs_task_cpu and ecs_task_memory are set to 256 (1/4 CPU) and 512 MB  respectively, which is veerry low. Fargate also seems to be very bad at vertical scaling (bursting) according to this issue, which would explain why our server doesn't scale fast enough to handle the 20 parallel plans.
The only other reason I see for using Fargate otherwise is if the atlantis server could somehow accomodate horizontal scaling. The ecs_service_desired_count parameter is set to 1 as default however, and autoscaling is turned off (don't even see a variable to enable it...). The complete example uses an ALB however, which seems to indicate horizontal scaling is happening??
Describe the solution you'd like.
I might be misunderstanding something, but I don't understand why atlantis is being run on ecs/fargate. I would like to see some more extensive (there is currently none, at all) discussion on the setting of the resource parameters and auto-scaling properties. Otherwise, it might make more sense to give the possibility to run this on an ec2 instance?

Atlantis itself does not have great support for horizontal/parallelized scaling and therefore the recommendation would be to:

Increase the task size resources (cpu and memory)
Look at optimizing how changes are detected and handled within Atlantis to avoid unnecessary plans on no-op changes (given you are using workspaces, I suspect you won't be able to benefit from this though)

In short, there isn't anything that this module can do to alleviate your issues at this time


Detect literals in alternations for regex strategy

Regexes such as 0|[1-9][0-9]* end up matching 0 half the time. It would be good if proptest could detect one of the cases will always match exactly one input (0) and match it exactly once instead of 50% of the executions. I guess a greedy version would be to detect other cases, but literals are good enough for me.

I just tripped over this myself -- 62 out of 261 test iterations with this strategy generated the empty string.  I do want the empty string tested, but I'd much rather have it emit 61 more cases probing the other alternatives.
        r#"(?sx:
                                      # empty string
        | [^0-9] .*                   # a non-digit, then anything
        | [0-9]+ [^0-9.] .*           # digits, non-(digit or dot), anything
        | [0-9]+ \. [0-9]* [^0-9] .*  # digits, dot, maybe more digits,
                                      # non-digit, anything
        )"#

There is a relatively easy workaround -- pull the fixed case(s) out to a separate, non-proptest test -- but then you lose the property of having your complete test strategy in one place (this regex can currently generate all the strings that my parser should reject).

I ended up doing this and later on just implementing arbitrary for my type but as you said it's a little bit away from being perfect.


Add stub release note for v1.1

Encourage developers to contribute to the changelog as they go.

:+1:

Just in case, I'd like to point out how jazzband documents releases in "How to decide for the version number?" (with BREAKING.FEATURE.FIX numbering). I made a similar comment in https://github.com/jazzband/django-pipeline/issues/515.
It would be great if we could stick to that.

@slafs you are right it could be 1.0.1 instead. I'll change it. Anyway we will decide the version number when doing the release.

Thanks.

Anyway we will decide the version number when doing the release.

Yes, of course.

Maybe we should also bump the version in configurations/__init__.py already - to 1.X-dev?

@blueyed Agreed! After every release the version should be changed. This way dev releases will be installed.

@blueyed good point, thanks. #139


Implement Internal Bar Strenght

Expected Behavior
Supports Internal Bar Strength (see talipp implementation).
We use this indicator in the GlobalEquityMeanReversionIBSAlpha, so we should use the new indicator there.
Refer to LEAN Engine > Contributions > Indicators.
Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[x] I have confirmed that this is not a duplicate issue by searching issues
[x] I have provided detailed steps to reproduce the issue

Strenght -> Strength


The single sign on service should use its own default ssl configuration.

Using the default SSL configuration for sso can cause issues when other services are using the same config.

woops, wrong project, this has nothing to do with this project.

not valid at the moment.


Add "dreamwidth.org" to Yes Porn Allow list

What is the site you want to add?
Dreamwidth
What kind of service are they?
Social Media
How does the site tend to treat NSFW content?
Allowed if it's legal under US law (ToS section XI). Marking enouraged but not mandatory.
Do you know of any risk to the site?
No
Any comment you want to add about the site?
'Slow' social media (this may be a pro or a con). Image hosting limited to on-site display only. Strong privacy features. Site is fully user-funded, and site owners have a strong commitment to protecting first amendment rights

Site owner here to confirm: as long as it's not child sex abuse material (CSAM) by US law, we allow all pornographic content, whether visual or written, and we have in fact turned up in court to help protect users' rights. We have very limited on-site photo hosting and don't allow hotlinking, so we're not a good fit for anyone who's looking for image or video hosting, but we have a large contingent of erotica writers, erotic roleplayers, and adult-content bloggers.

Finally got around to adding this to the list. Definitely a good addition!


Optional libpeas causes the build to fail on Ubuntu 20.04

Ubuntu 20.04 currently has libpeas-dev (1.26.0-2) in its repos. When trying to build with this installed, ninja will fail giving this error:
➜  ahoviewer git:(master) meson build -Dbuildtype=release
The Meson build system
Version: 0.55.1
Source dir: /home/anon/ahoviewer
Build dir: /home/anon/ahoviewer/build
Build type: native build
Project name: ahoviewer
Project version: 1.6.5-133-g77bc172
C compiler for the host machine: cc (gcc 9.3.0 "cc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0")
C linker for the host machine: cc ld.bfd 2.34
C++ compiler for the host machine: c++ (gcc 9.3.0 "c++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0")
C++ linker for the host machine: c++ ld.bfd 2.34
Host machine cpu family: x86_64
Host machine cpu: x86_64
Run-time dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.29.1)
Run-time dependency glibmm-2.4 found: YES 2.64.2
Run-time dependency sigc++-2.0 found: YES 2.10.2
Run-time dependency gtk+-3.0 found: YES 3.24.20
Run-time dependency gtkmm-3.0 found: YES 3.24.2
Run-time dependency x11 found: YES 1.6.9
Run-time dependency libconfig++ found: YES 1.5
Run-time dependency libxml-2.0 found: YES 2.9.10
Run-time dependency libcurl found: YES 7.68.0
Dependency openssl found: NO found 1.1.1f but need: '<1.1.0'
Found CMake: /usr/bin/cmake (3.16.3)
Run-time dependency openssl found: NO (tried pkgconfig and cmake)
Run-time dependency gstreamer-1.0 found: NO (tried pkgconfig and cmake)
Run-time dependency gstreamer-audio-1.0 found: NO (tried pkgconfig and cmake)
Run-time dependency gstreamer-video-1.0 found: NO (tried pkgconfig and cmake)
Run-time dependency libpeas-1.0 found: YES 1.26.0
Run-time dependency libsecret-1 found: YES 0.20.3
Run-time dependency libunrar found: NO (tried pkgconfig and cmake)
Library libunrar found: NO
Has header "dll.hpp" : NO
Has header "dll.hpp" : NO
Run-time dependency libzip found: YES 1.5.1
Run-time dependency libnsgif found: NO (tried pkgconfig and cmake)
Found pkg-config: /usr/bin/pkg-config (0.29.1)
Build-time dependency gobject-introspection-1.0 found: YES 1.64.1
Program g_ir_scanner found: YES (/usr/bin/g-ir-scanner)
Program g_ir_compiler found: YES (/usr/bin/g-ir-compiler)
Configuring config.h using configuration
Build targets in project: 7

Found ninja-1.10.0 at /usr/bin/ninja
➜  ahoviewer git:(master) cd build
➜  build git:(master) ninja
[41/50] Compiling C++ object src/ahoviewer.p/settings.cc.o
FAILED: src/ahoviewer.p/settings.cc.o
c++ -Isrc/ahoviewer.p -Isrc -I../src -I../ext/entities -I../ext/date/include -I../ext/json -I../ext/libnsgif -I/usr/include/glibmm-2.4 -I/usr/lib/x86_64-linux-gnu/glibmm-2.4/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/sigc++-2.0 -I/usr/lib/x86_64-linux-gnu/sigc++-2.0/include -I/usr/include/gtkmm-3.0 -I/usr/lib/x86_64-linux-gnu/gtkmm-3.0/include -I/usr/include/atkmm-1.6 -I/usr/include/gtk-3.0/unix-print -I/usr/include/gdkmm-3.0 -I/usr/lib/x86_64-linux-gnu/gdkmm-3.0/include -I/usr/include/giomm-2.4 -I/usr/lib/x86_64-linux-gnu/giomm-2.4/include -I/usr/include/pangomm-1.4 -I/usr/lib/x86_64-linux-gnu/pangomm-1.4/include -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib/x86_64-linux-gnu/dbus-1.0/include -I/usr/include/gio-unix-2.0 -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/fribidi -I/usr/include/harfbuzz -I/usr/include/atk-1.0 -I/usr/include/cairomm-1.0 -I/usr/lib/x86_64-linux-gnu/cairomm-1.0/include -I/usr/include/pixman-1 -I/usr/include/uuid -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/libxml2 -I/usr/include/x86_64-linux-gnu -I/usr/include/libpeas-1.0 -I/usr/include/gobject-introspection-1.0 -I/usr/include/libsecret-1 -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++17 -O3 -pthread -DHAS_REMOTE_API=1 -DHAS_UNCAUGHT_EXCEPTIONS=1 -MD -MQ src/ahoviewer.p/settings.cc.o -MF src/ahoviewer.p/settings.cc.o.d -o src/ahoviewer.p/settings.cc.o -c ../src/settings.cc
../src/settings.cc: In member function ‘void AhoViewer::SettingsManager::load_keybindings()’:
../src/settings.cc:450:54: error: no match for ‘operator[]’ (operand types are ‘libconfig::Setting’ and ‘std::string’ {aka ‘std::__cxx11::basic_string<char>’})
  450 |                                 keys[i.first.c_str()][p->get_action_name()].c_str();
      |                                                      ^
In file included from ../src/settings.h:5,
                 from ../src/settings.cc:1:
/usr/include/libconfig.h++:223:13: note: candidate: ‘libconfig::Setting& libconfig::Setting::operator[](const char*) const’
  223 |   Setting & operator[](const char *name) const;
      |             ^~~~~~~~
/usr/include/libconfig.h++:223:36: note:   no known conversion for argument 1 from ‘std::string’ {aka ‘std::__cxx11::basic_string<char>’} to ‘const char*’
  223 |   Setting & operator[](const char *name) const;
      |                        ~~~~~~~~~~~~^~~~
/usr/include/libconfig.h++:224:13: note: candidate: ‘libconfig::Setting& libconfig::Setting::operator[](int) const’
  224 |   Setting & operator[](int index) const;
      |             ^~~~~~~~
/usr/include/libconfig.h++:224:28: note:   no known conversion for argument 1 from ‘std::string’ {aka ‘std::__cxx11::basic_string<char>’} to ‘int’
  224 |   Setting & operator[](int index) const;
      |                        ~~~~^~~~~
[49/50] Compiling C++ object src/ahoviewer.p/mainwindow.cc.o
ninja: build stopped: subcommand failed.
➜  build git:(master)

By removing libpeas-dev from my system, the software was able to compile properly.

Those errors are actually caused by libconfig (only occurs in libconfig 1.5 as far as I remember).
I need to change the plugin settings loading code to match the other parts to be compatible with libconfig 1.5.

Those errors are actually caused by libconfig (only occurs in libconfig 1.5 as far as I remember).
I need to change the plugin settings loading code to match the other parts to be compatible with libconfig 1.5.

This should fix the issue, but I have not tested to make sure.


init asks for var.container_repo

Running ./init asks for the var.container_repo variable, which looks to be docker.io/timfpark. What's that repo for?

that was while deploying acs-engine.

Ooops - yes, i see that now.  Fixed in change 56d3828.


Added Speedy Labs LightFox Config

Speedy Labs LightFox
Technically still requires GP2040-CE to invert the main 12 buttons, which is currently not implemented.

Good start to things.
I would like to wait until the device is finished and the code needed to run it has been merged into main.
Are you planning to open source the device?

Controllers are closed source while they are being actively sold.  I release files when a model is no longer being sold/supported by me.

Good start to things.
I would like to wait until the device is finished and the code needed to run it has been merged into main.
Are you planning to open source the device?

Controllers are closed source while they are being actively sold. I release files when a model is no longer being sold/supported by me.

This puts us in an odd spot.
For now we have paused any new inclusion of configs while we work through new submission guidelines.
Is there an ETA on the lifecycle for the product before its open sourced?


Better URL comparison for metrics

Codecov Report

Merging #206 into v2-master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##           v2-master     #206      +/-   ##
=============================================
- Coverage      53.17%   53.15%   -0.02%
=============================================
  Files            918      918
  Lines          25286    25286
  Branches        4335     4335
=============================================
- Hits           13447    13442       -5
- Misses         11839    11844       +5


Update README.md

Added PrefixConversion.java file
Description
Please include a summary of the changes you have made in regard with this issue.
Fixes #274 (issue number)
Type of change
Please delete options that are not relevant

[x] This change requires a documentation update

Checklist:

[x] I have read the contribution guidelines before making this PR.
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented in my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation (Language folder or any other readme instructed to)
[x] My changes generates no new warnings
[x] I have added tests that prove my fix is effective or that my feature works

You need not make separate PR for this. Add this readme commit in previous PR only

Okay, sorry I didnt know about that it can be done there.
Updating there now.


Error: error pulling "": unable to pull or load image on a root-less container

Problem: podman load -i rhel6-6.tar fails in non-root user environment
guest2@host:/home $podman load -i rhel6-6.tar
Getting image source signatures
Copying blob f65fa38dc372 done
Copying config 594a9a79ca done
Writing manifest to image destination
Storing signatures
ERRO[0006] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument
ERRO[0006] Error pulling image ref rhel6-6.tar: Error committing the finished image: error adding layer with blob "sha256:f65fa38dc3729ba856434b2a8103c041cb5b06465b7884bb1eaa4c308378a3c4": ApplyLayer exit status 1 stdout:  stderr: invalid argument
Failed
ERRO[0006] Error pulling image ref rhel6-6.tar: Error determining manifest MIME type for dir:rhel6-6.tar: open rhel6-6.tar/manifest.json: not a directory
Failed
Error: error pulling "": unable to pull dir:rhel6-6.tar: unable to pull image: Error determining manifest MIME type for dir:rhel6-6.tar: open rhel6-6.tar/manifest.json: not a directory
Note: The same command works perfectly fine as root user.
root@host:home # podman load -i rhel6-6.tar
Getting image source signatures
Copying blob f65fa38dc372 done
Copying config 594a9a79ca done
Writing manifest to image destination
Storing signatures
Loaded image(s): rhel6:6.8-65
podman version 1.4.4
cat /etc/*id
guest2:165536:65536
guest2:165536:65536

Any chance this could be re-tried with a more recent Podman? RHEL/CentOS 7.8 and 8.2 should have a 1.6.x release of Podman available.

Where did rhel6-6.tar come from?


Cannot find module index.js

https://gyazo.com/7fe8d0a9c1d40681a20d9eaef438c6c3
This is the error that happens
Anyone here that knows a fix?
index.js is in src by the way.

then do
sudo node src/index.js

dont worry, i fixed it. by Reinstalling everything.